Visual servo control of intelligent wheelchair mounted robotic arm

Wheelchair Mounted Robotic Arms (WMRA) can be used by people with severe motor skill impairment, such as SMA (Spinal Muscular Atrophy), Cerebral Palsy etc…, in order to achieve daily life tasks. Many of those systems have been presented in literature and are available on the market but they are really expensive and bulky.

A wheelchair-mounted robotic arm (WMRA) was designed and built to meet the needs of mobility-impaired persons with limitations of upper extremities, and to exceed the capabilities of current devices of this type. The mechanical design incorporates DC servo drive, with actuator hardware at each individual joint, allowing reconfigurable link lengths. It has seven degrees of freedom and uses a side mount on a power wheelchair. The control system allows coordinated Cartesian control, and offers expandability for future research, such as coordinated motion with the wheelchair itself. This paper discusses the current state of the art in WMRAs; describes the design goals and user requirements for this device; explains the component selection process; discusses details of the mechanical design, electrical system and low-level controller; covers manufacturing concerns; and describes the testing of the completed arm. Further improvements are also suggested

This paper aims to develop a visual servo control of a robotic manipulator for cherry tomato harvesting. In the robotic manipulator, an RGB-depth camera was mounted to the end effector to acquire the poses of the target cherry tomatoes in space. The eye-in-hand-based visual servo controller guides the end effector to implement eye–hand coordination to harvest the target cherry tomatoes, in which a hybrid visual servo control method (HVSC) with the fuzzy dynamic control parameters was proposed by combining position-based visual servo (PBVS) control and image-based visual servo (IBVS) control for the tradeoff of both performances. In addition, a novel cutting and clipping integrated mechanism was designed to pick the target cherry tomatoes. The proposed tomato-harvesting robotic manipulator with HVSC was validated and evaluated in a laboratory testbed based on harvesting implementation. The results show that the developed robotic manipulator using HVSC has an average harvesting time of 9.40 s/per and an average harvesting success rate of 96.25% in picking cherry tomatoes.

Visual servo control rules that refer to the control methods of robot motion planning using image data acquired from the camera mounted on the robot have been widely applied to the motion control of robotic arms or mobile robots. The methods are usually classified as image-based visual servo, position-based visual servo, and hybrid visual servo (HVS) control rules. Mobile manipulation enhances the working range and flexibility of robotic arms. However, there is little work on applying visual servo control rules to the motion of the whole mobile manipulation robot. We propose an HVS motion control method for a mobile manipulation robot which combines a six-degree-of-freedom (6-DOF) robotic arm with a nonholonomic mobile base. Based on the kinematic differential equations of the mobile manipulation robot, the global Jacobian matrix of the whole robot is derived, and the HVS control equation is derived using the whole Jacobian matrix combined with position and visual image information. The distance between the gripper and target is calculated through the observation of the marker by a camera mounted on the gripper. The differences between the positions of the markers' feature points and the expected positions of them in the image coordinate system are also calculated. These differences are substituted into the control equation to obtain the speed control law of each degree of freedom of the mobile manipulation robot. To avoid the position error caused by observation, we also introduce the Kalman filter to correct the positions and orientations of the end of the manipulator. Finally, the proposed algorithm is validated on a mobile manipulation platform consisting of a Bulldog chassis, a UR5 robotic arm, and a ZED camera.

For the automatic docking problem of a Mecanum-wheeled omnidirectional intelligent wheelchair/bed system, a visual servo control method which is based on position feedback is proposed in this paper. When it is docked with the auxiliary bed, the intelligent wheelchair firstly collects and identifies special landmark on the auxiliary bed through the vehicle vision system; Then the relative pose between the wheelchair and the auxiliary bed is determined according to the transformation between the camera coordinate system and the bed coordinate system based on an artificial landmark; Finally, the intelligent wheelchair is controlled to travel from the initial position to the desired position by position-based visual servo (PBVS) control method and completes docking task between the intelligent wheelchair and the auxiliary bed. The implementation of visual docking control for the intelligent wheelchair is based on the Robotic Operating System (ROS) platform. Experimental results are presented to validate the practicality and effectiveness of the proposed method in this paper.

In a household setting, a wheelchair-mounted robotic arm (WMRA) can be useful for assisting elderly and disabled individuals. However, the current WMRA can only perform movement and grasping tasks through joystick remote control. This method results in low efficiency due to poor coordination between the mobile platform and the robotic arm as well as the numerous operational steps required. To improve the efficiency and success rate of the robot in task execution, this paper proposes a parking location optimization method that combines the occupied grid map (OGM) and the inverse reachability map (IRM). Firstly, the SLAM algorithm is used to collect environment information, which is then stored in the form of an occupied grid map. The robotic arm workspace is then gridded, and the inverse reachability map is calculated based on the grasping pose of the target object. Finally, the optimal position of the mobile platform is obtained by comparing the optimal location point in the inverse reachability map and the obstacle information in the occupied grid map. This process achieves base placement optimization based on the grasping pose. The experimental results demonstrate that this method reduces the user operation time by 97.31% and overall task completion time by 40.57% when executing household environment tasks compared with the joystick control, increasing the range of executable tasks compared with the algorithm of the EL-E robot and reducing task completion time by 23.48% for the same task. This paper presents a parking location optimization method that can improve the grasping efficiency of the robotic arm and achieve parking location position selection for the WMRA in a household environment.

At the beginning of its application, arm-robot manipulator worked blindly in completing the assigned task, and along with the improvement of camera technology as it is getting cheaper and smaller, the application of visual sensor lead to visual servoing control which utilized target detection as the input to control/move the end-effector of an arm-robot. The application of camera in arm-robot manipulator is divided into eye-in-hand where a camera is mounted to the robot, and eye-to-hand where the camera is fixed elsewhere. Visual control involves kinematics modeling and target position in image plane provided by image processing to ensure the motion accuracy in grasping and harvesting the fruit. The end-effector movement for harvesting relies heavily on the accuracy of target position detection that can be obtained by image processing method. This paper is a review study that discusses the design of visual servoing control of an arm-robot manipulator applied in agriculture starting from kinematics and inverse kinematics for controlling the motion and trajectory of the robot, and image processing for target detecting. An example of an image processing method is given by using Edge Detection method for detecting a dragon fruit simulated in SCILAB.

Visual servo control systems based on Kalman filter (KF) is susceptible to noise interference, the initialization of the Jacobi matrix is difficult, and the observation value of the Jacobian matrix is not accurate. In order to address these problems, we proposed a robust KF algorithm with long short-term memory (LSTM) for an image-based visual servo control system and applied the system to an uncalibrated image-based visual servo (IBVS) control system to estimate the filtering gain error, state estimation error, and the observation error, which were then used for online training in LSTM. The visual servo control system controls the motion of the manipulator, and simultaneously updates the LSTM network. Therefore, the Jacobian matrix obtained using LSTM was employed to estimate the state volume of the robust KF, which constitutes a circulatory system, and the complementary effect was realized. The method was applied to a six-degrees-of-freedom manipulator of the eye-in-hand model to conduct experiments. The simulation results indicate that the proposed visual servo control system has strong anti-noise interference capability. Furthermore, it facilitates Jacobian matrix initialization and has high observation accuracy for the Jacobian matrix.

There has been significant progress in bringing commercially-viable wheelchair mounted robotic arms (WMRA) into the marketplace in the past 30 years. This paper focuses on kinematic analysis and evaluation of such robotic arms. It addresses the kinematics of the WMRA with respect to its ability to reach common positions while performing activities of daily living (ADL). A procedure is developed for the kinematic analysis and evaluation of a wheelchair mounted robotic arm. In addition to developing the analytical procedure, the manipulator is evaluated, and design recommendations and insights are obtained. Current commercially-available wheelchair mountable robotic manipulators have been designed specifically for use in rehabilitation robotics. In an effort to evaluate two commercial manipulators, the procedure for kinematic analysis is applied to each manipulator. Design recommendations with regard to each device are obtained. This method will benefit the researchers by providing a standardized procedure for kinematic analysis of WMRAs that is capable of evaluating independent designs.

This paper proposes a visual servoing controller design based on Barrier Lyapunov function for a picking system. Visual servoing uses feedback data provided by the camera to control the movement of a picking system in a closed loop system. Visual servoing requires an object in the field of view of the camera in order to control the picking system. To improve the visual servoing controller, the image-based visual servoing and the position-based visual servoing are presented. To apply this method an offline trajectory is developed to perform the image-based visual servoing and the position-based visual servoing tasks for the picking system. Two different control approaches i.e. the visual servoing controller with the limit orientation using the Barrier Lyapunov function and the visual servoing controller with a quadratic Lyapunov function are presented. The proof of asymptotic stability is presented and simulation results from two visual servoing controllers are presented to verify the effectiveness of the proposed controller.

Wheelchair-mounted robotic arms support people with upper extremity disabilities with various activities of daily living (ADL). However, the associated cost and the power consumption of responsive and adaptive assistive robotic arms contribute to the fact that such systems are in limited use. Neuromorphic spiking neural networks can be used for a real-time machine learning-driven control of robots, providing an energy efficient framework for adaptive control. In this work, we demonstrate a neuromorphic adaptive control of a wheelchair-mounted robotic arm deployed on Intel’s Loihi chip. Our algorithm design uses neuromorphically represented and integrated velocity readings to derive the arm’s current state. The proposed controller provides the robotic arm with adaptive signals, guiding its motion while accounting for kinematic changes in real-time. We pilot-tested the device with an able-bodied participant to evaluate its accuracy while performing ADL-related trajectories. We further demonstrated the capacity of the controller to compensate for unexpected inertia-generating payloads using online learning. Videotaped recordings of ADL tasks performed by the robot were viewed by caregivers; data summarizing their feedback on the user experience and the potential benefit of the system is reported.

This article deals with the depth observability problem of a robot visual system with a moving camera. In the visual system, the unknown depth of a feature point is estimated from the input of the camera velocity and the output of the image of the feature point. Although it is well known that the linear velocity of the camera must satisfy some constraints for successful depth estimation, this proposes a criterion to measure the performance of the depth estimation, which is a heuristic extension from an estimation result of a linear system. This performance criterion depends on both the image position and the linear velocity of the camera. Some simulation and experiment examples demonstrate and verify the proposed performance criterion. Furthermore, this criterion is used to develop a new visual servo control scheme that has good performance in both the depth estimation and the visual control. This control scheme is also verified by a simulation example. © 2001 John Wiley & Sons, Inc.

This paper aims to develop agricultural robots that can be applied to greenhouse crop harvesting by using a hybrid visual servo control method (HVSC). In the research, a depth camera was used to acquire the posture of the tomato in three-dimensional space, and visual servo control can be carried out for the tomato growing at different angles in practice. Different visual servo control methods are also discussed, including the Position-Based Visual Servo (PBVS), Image-Based Visual Servo (IBVS) and the proposed HVSC with the fuzzy dynamic control parameters. Characteristics of different visual servo control methods were discussed, and then applied to the actual harvesting. The results show that the hybrid visual servo control developed in this research has an average harvesting time of 9.40s/per and an average harvesting success rate of 96.25% for cherry tomato.

A modified image-based visual servo controller with hybrid camera configuration for robust robotic grasping

Wheelchair-Mounted Robotic Arms have been used to help impaired people to reach objects and perform essential activities in an autonomous way. Different available models are presented in this paper and a simple design is proposed to improve the kinematic performances of the integrated system in order to allow the user to increase its capability of interaction with home environment. To this end, a linear drive has been added to the Raptor model in order to move along the wheelchair. The benefit of the proposed development has been proved with a kinematic performance assessment procedure, which has analyzed critical points in the 3D space, providing 26% increase in performance with respect to the existing solution.