Vision-based Robot Research Articles

Optimization modeling and application of machine vision-based robot roller hemming for autobody panels with adhesive

Optimization modeling and application of machine vision-based robot roller hemming for autobody panels with adhesive

Design, Modeling, and Experimental Validation of a Vision-Based Table Tennis Juggling Robot

This paper develops a new vision-based robot customized for table tennis juggling tasks. Specifically, the robot is equipped with two industrial cameras operating as a sensing system. An image-processing algorithm is proposed that allows the robot to balance a table tennis ball while controlling its bounce height. The robot adopts a parallel structure design, and the end effector employs three ball joints to increase the degree of freedom (DOF) of the parallel mechanism. In addition, we design a control scheme explicitly customized for this robotic system. Extensive real-time experiments are performed to show the effectiveness of the juggling robot at different jumping heights. Furthermore, the ability to consistently maintain a fixed preset bounce height is demonstrated. These experimental results confirm the efficacy of the developed robotic system.

Robot motion visual measurement based on RANSAC and weighted constraints method

Precision of the coordinate transformation model parameters is crucial for the accuracy of the vision-based robot spatial motion measurement method. In this work, an optimization algorithm integrating RANSAC and iterative weighted singular value decomposition (IR-SVD) is proposed for improving the coordinate transformation model solution precision, aiming at enhancing the spatial motion measurement accuracy of the binocular vision system. Considering noises existing in reference transformation point pairs, the RANSAC algorithm is introduced to filter the raw measurement point pairs and extract inliers, thereby eliminating potential gross errors and realizing the cluster of advantageous points. An enhanced SVD method based on iterative weighted constraints is proposed to substitute traditional SVD. After calculating the coordinate transformation model parameters, the measurement errors of inliers are solved synchronously, and the weights are reallocated in light of the measurement errors value, subsequently recalculating the coordinate transformation model parameters repeatedly until the errors converge. The validation experiments are conducted on the self-built three-degree-of-freedom rectangular coordinate robot platform. The experimental results of discrete point-to-point motion and continuous trajectory motion measurement show that the proposed method can improve the coordinate transformation model parameters solution accuracy effectively, comparing with the traditional SVD method. Comparative experiment with existing commonly used coordinate transformation methods including Quaternion and iterative closest point indicates that the proposed method exhibits the best applicability and minimal errors in robot motion visual measurement. Both accuracy of the coordinate transformation model solution and the visual system’s motion measurement are enhanced with this newly-proposed, optimized method.

Vision-based robotics using open FPGAs

Robotics increasingly provides practical applications for society, such as manufacturing, autonomous driving, robot vacuum cleaners, robots in logistics, drones for inspection, etc. Typical requirements in this field are fast response time, low power consumption, parallelism, and flexibility. According to these features, FPGAs are a suitable computing substrate for robots. A few vendors have dominated the FPGA market with their proprietary tools and hardware devices, resulting in fragmented ecosystems with few standards and little interoperation. New and complete open toolchains for FPGAs are emerging from the open-source community. This article presents an open-source library of Verilog modules useful for vision-based robots, including reusable image processing blocks for perception and reactive control blocks. This library has been developed using open tools, but its Verilog modules are fully compatible with any proprietary toolchain. In addition, three applications with a real robot and open FPGAs have been developed for experimental validation using this library. In the last application, the mobile robot successfully follows a colored object using two low-cost cameras (to increase the robot’s field of view) and includes a third camera on top of a servo-driven turret for tracking a second independent object while following the first one in parallel. Resource consumption of all applications has been measured and compared with state-of-the-art proprietary toolchains, revealing that reconfigurable computing with open FPGAs using open tools is now an attractive alternative to designing and creating intelligent vision-based robotic applications using vendor-dependent proprietary tools and FPGAs.

Convolutional Neural Network-Based Robot Control for an Eye-in-Hand Camera

In past decades, much progress has been obtained in vision-based robot control theories with traditional image processing methods. With the advances in deep-learning-based methods, convolutional neural network (CNN) has now replaced the traditional image processing methods for object detection and recognition. However, it is not clear how the CNN-based methods can be integrated into robot control theories in a stable and predictable manner for object detection and tracking, especially when the aspect ratio of the object is unknown and also varies during manipulation. In this article, we develop a vision-based control method for robots with an eye-in-hand configuration, which can be directly integrated with existing CNN-based object detectors. The task variables are generated based on parameters of the bounding box from the output of any real-time CNN object detector such as you only look once (Yolo). To address the chattering problem of bounding box, long short-term memory (LSTM) is used to provide smoothed bounding box information. A vision-based controller is then proposed following task-space motion control design formulation in order to keep the object of unknown aspect ratio in the center of field of view of the camera. The stability of the overall closed-loop control system is analyzed rigorously using the Lyapunov-like approach. Experimental results are presented to illustrate the performance of the proposed CNN-based robot controller.

A novel robot hand-eye calibration method to enhance calibration accuracy based on the POE model

Hand-eye calibration is crucial for vision-based robots using open-loop visual control. Based on the product of exponentials model, this study introduces a novel hand-eye calibration algorithm to improve calibration accuracy. As opposed to the traditional method of solving the AX = XB type in the hand-eye calibration using the marker, we transform the solution into an AX = B type without requiring accurate camera parameters. Firstly, the nominal marker pose in the robot base is described by the nominal kinematic parameters of the robot, camera, and end-effector-to-camera. The actual marker pose is obtained by physically contacting the marker with the probe. Subsequently, the error parameter of the hand-eye pose is defined as linear to the error between the nominal and actual marker poses and can be solved by an iterative least-squares algorithm. Finally, the calibrated hand-eye pose is acquired by multiplying its nominal pose and the exponential mapping of the error parameter. The simulations and real experiments illustrate the effectiveness and stability of the proposed algorithm. Noteworthy, the mean position accuracy of the vision-based robot is improved nearly 20 times after calibration. Furthermore, the comparison experiment demonstrates that the proposed algorithm is preferable for improving hand-eye calibration accuracy.

Vision-Based Robot Arm Control Interface for Retrieving Objects from the Floor

Approximately half of the patients with spinal cord injuries in Japan have a cervical spinal cord injury. Owing to the trunk dysfunction, patients with high-level spinal cord injuries have particular difficulty when searching for or picking up objects from the floor. Recently, welfare robot arms have been developed to help such individuals increase self-reliance. In this study, we propose an operating system that includes an eye-in-hand system with a touchscreen interface for grasping objects from the floor and delivering them to the individual. In the proposed method, the visual information of the target object is shown on a touchscreen interface. The patient specifies the target position for the robot arm by drawing a line on the target object on the interface. We conducted an experiment to compare the proposed interface with an on-screen joystick to demonstrate the proposed system’s efficiency and its ability to reduce physical burden. The results show that the proposed method is both quicker to use and effectively reduces the physical burden on the user compared to the conventional method.

Vision-based Robot Arm Grasping in Medical region

This paper proposes a vision-based robot arm grasping system for automatically sorting various and complicated medicines and helping take care of patients, etc in medical area. This system could successfully identify and locate the target objects in a complex and messy environment. To achieve accurately location, the system uses several cameras to recognition and estimation. The cameras cover various kinds, visual, laser, and mechanical sensors are all included.The computer will process the data and send instructions to robot arms for grasping. The experiment shows that the vision algorithm will efficiently help to sort medicines and other works. The robotic arms are fully covered by leather which makes it more safe and the internal lube and anti-slip pads on the gripper make it more flexible and smooth when grasping objects. During the COVID-19 pandemic, the robotic arm adjusted from industries to hospital so that it plays an important role in Nucleic acid testing, drug classification, disinfection. Moreover, it could help not only nurses to monitor the patients’ health but also patients to take better care of themselves.

Rethinking causality-driven robot tool segmentation with temporal constraints.

Vision-based robot tool segmentation plays a fundamental role in surgical robots perception and downstream tasks. CaRTS, based on a complementary causal model, has shown promising performance in unseen counterfactual surgical environments in the presence of smoke, blood, etc. However, CaRTS requires over 30 iterations of optimization to converge for a single image due to limited observability. To address the above limitations, we take temporal relation into consideration and propose a temporal causal model for robot tool segmentation on video sequences. We design an architecture named Temporally Constrained CaRTS (TC-CaRTS). TC-CaRTS has three novel modules to complement CaRTS-temporal optimization pipeline, kinematics correction network, and spatial-temporal regularization. Experiment results show that TC-CaRTS requires fewer iterations to achieve the same or better performance as CaRTS on different domains. All three modules are proven to be effective. We propose TC-CaRTS, which takes advantage of temporal constraints as additional observability. We show that TC-CaRTS outperforms prior work in the robot tool segmentation task with improved convergence speed on test datasets from different domains.

A robust and simple collision avoidance approach for pick and place robot applications

Purpose The aim of this study is to create a robust and simple collision avoidance approach based on quaternion algebra for vision-based pick and place applications in manufacturing industries, specifically for use with industrial robots and collaborative robots (cobots). Design/methodology/approach In this study, an approach based on quaternion algebra is developed to prevent any collision or breakdown during the movements of industrial robots or cobots in vision system included pick and place applications. The algorithm, integrated into the control system, checks for collisions before the robot moves its end effector to the target position during the process flow. In addition, a hand–eye calibration method is presented to easily calibrate the camera and define the geometric relationships between the camera and the robot coordinate systems. Findings This approach, specifically designed for vision-based robot/cobot applications, can be used by developers and robot integrator companies to significantly reduce application costs and the project timeline of the pick and place robotics system installation. Furthermore, the approach ensures a safe, robust and highly efficient application for robotics vision applications across all industries, making it an ideal solution for various industries. Originality/value The algorithm for this approach, which can be operated in a robot controller or a programmable logic controller, has been tested as real-time in vision-based robotics applications. It can be applied to both existing and new vision-based pick and place projects with industrial robots or collaborative robots with minimal effort, making it a cost-effective and efficient solution for various industries.

Research on vision-based robot planar motion measurement method

Research on vision-based robot planar motion measurement method

Bilateral Cross-Modal Fusion Network for Robot Grasp Detection

In the field of vision-based robot grasping, effectively leveraging RGB and depth information to accurately determine the position and pose of a target is a critical issue. To address this challenge, we proposed a tri-stream cross-modal fusion architecture for 2-DoF visual grasp detection. This architecture facilitates the interaction of RGB and depth bilateral information and was designed to efficiently aggregate multiscale information. Our novel modal interaction module (MIM) with a spatial-wise cross-attention algorithm adaptively captures cross-modal feature information. Meanwhile, the channel interaction modules (CIM) further enhance the aggregation of different modal streams. In addition, we efficiently aggregated global multiscale information through a hierarchical structure with skipping connections. To evaluate the performance of our proposed method, we conducted validation experiments on standard public datasets and real robot grasping experiments. We achieved image-wise detection accuracy of 99.4% and 96.7% on Cornell and Jacquard datasets, respectively. The object-wise detection accuracy reached 97.8% and 94.6% on the same datasets. Furthermore, physical experiments using the 6-DoF Elite robot demonstrated a success rate of 94.5%. These experiments highlight the superior accuracy of our proposed method.

A Comprehensive Study of 3-D Vision-Based Robot Manipulation.

Robot manipulation, for example, pick-and-place manipulation, is broadly used for intelligent manufacturing with industrial robots, ocean engineering with underwater robots, service robots, or even healthcare with medical robots. Most traditional robot manipulations adopt 2-D vision systems with plane hypotheses and can only generate 3-DOF (degrees of freedom) pose accordingly. To mimic human intelligence and endow the robot with more flexible working capabilities, 3-D vision-based robot manipulation has been studied. However, this task is still challenging in the open world especially for general object recognition and pose estimation with occlusion in cluttered backgrounds and human-like flexible manipulation. In this article, we propose a comprehensive analysis of recent progress about the 3-D vision for robot manipulation, including 3-D data acquisition and representation, robot-vision calibration, 3-D object detection/recognition, 6-DOF pose estimation, grasping estimation, and motion planning. We then present some public datasets, evaluation criteria, comparisons, and challenges. Finally, the related application domains of robot manipulation are given, and some future directions and open problems are studied as well.

A Comprehensive Review of Vision-Based Robotic Applications: Current State, Components, Approaches, Barriers, and Potential Solutions

Being an emerging technology, robotic manipulation has encountered tremendous advancements due to technological developments starting from using sensors to artificial intelligence. Over the decades, robotic manipulation has advanced in terms of the versatility and flexibility of mobile robot platforms. Thus, robots are now capable of interacting with the world around them. To interact with the real world, robots require various sensory inputs from their surroundings, and the use of vision is rapidly increasing nowadays, as vision is unquestionably a rich source of information for a robotic system. In recent years, robotic manipulators have made significant progress towards achieving human-like abilities. There is still a large gap between human and robot dexterity, especially when it comes to executing complex and long-lasting manipulations. This paper comprehensively investigates the state-of-the-art development of vision-based robotic application, which includes the current state, components, and approaches used along with the algorithms with respect to the control and application of robots. Furthermore, a comprehensive analysis of those vision-based applied algorithms, their effectiveness, and their complexity has been enlightened here. To conclude, there is a discussion over the constraints while performing the research and potential solutions to develop a robust and accurate vision-based robot manipulation.

Research on Intelligent Robot Point Cloud Grasping in Internet of Things.

The development of Internet of Things (IoT) technology has enabled intelligent robots to have more sensing and decision-making capabilities, broadening the application areas of robots. Grasping operation is one of the basic tasks of intelligent robots, and vision-based robot grasping technology can enable robots to perform dexterous grasping. Compared with 2D images, 3D point clouds based on objects can generate more reasonable and stable grasping poses. In this paper, we propose a new algorithm structure based on the PointNet network to process object point cloud information. First, we use the T-Net network to align the point cloud to ensure its rotation invariance; then we use a multilayer perceptron to extract point cloud characteristics and use the symmetric function to get global features, while adding the point cloud characteristics attention mechanism to make the network more focused on the object local point cloud. Finally, a grasp quality evaluation network is proposed to evaluate the quality of the generated candidate grasp positions, and the grasp with the highest score is obtained. A grasping dataset is generated based on the YCB dataset to train the proposed network, which achieves excellent classification accuracy. The actual grasping experiments are carried out using the Baxter robot and compared with the existing methods; the proposed method achieves good grasping effect.

Research Status of Robot Grab Detection Based on Vision

With the progress of science and technology and the continuous improvement of living standards, robots are more and more widely used in life and production, and robot grasping technology is also constantly improving. In practical application, accurate grasp detection of target objects is an important part of robot grasping task. In this paper, the parallel two-fingered gripper is used as the end of the robot arm's grasping, and the research status of grasping detection, which is the key part in the grasping process of the robot arm based on vision, is summarized. The 2D planar grasping and 6-DOF spatial grasping are compared and analyzed in detail. At the same time, it also summarizes the commonly used evaluation indexes of capturing data sets and capturing detection, and points out the challenges faced by vision-based robot capturing and the future direction of solving these challenges.

Adaptive ORB feature detection with a variable extraction radius in RoI for complex illumination scenes

Feature detection is a crucial technique for a vision navigation system to estimate robot pose according to natural landmarks. It is difficult for the existing feature detection techniques to balance the feature quality, processing time and robustness for a vision-based robot in complex workspaces. An adaptive Oriented fast and Rotated Brief (ORB) feature detection method with a variable extraction radius in Region of Interest (RoI) is proposed to deal with these problems. Firstly, the original camera image is processed by means of the Laplace transform of Gaussian (LTOG) pyramid and the grayscale centroid method, in order to obtain the rotation and scale invariance for ORB features. Then, a RoI segmenting technique is developed to locate the image areas that contain potential ORB features due to obvious grayscale variation. Thirdly, the ORB features are extracted in RoIs by using a set of variable-radius templates, adaptive to different illumination conditions. Finally, a number of feature detection and robot localization experiments are conducted on a vision-based robot prototype in different scenes under complex illumination. The experimental results verify that the RoI segmenting technique can correctly preserve the grayscale-varying regions to search ORB features with scattered distribution but excluding the irrelevant areas to suppress feature noises, while the variable-radius template extraction method can detect more feature inliers in complex workspaces. Therefore, our adaptive ORB method can outperform other commonly-used algorithms in accuracy, efficiency and robustness.

Autonomous Robotic Manipulation: Real-Time, Deep-Learning Approach for Grasping of Unknown Objects

Recent advancement in vision-based robotics and deep-learning techniques has enabled the use of intelligent systems in a wider range of applications requiring object manipulation. Finding a robust solution for object grasping and autonomous manipulation became the focus of many engineers and is still one of the most demanding problems in modern robotics. This paper presents a full grasping pipeline proposing a real-time data-driven deep-learning approach for robotic grasping of unknown objects using MATLAB and convolutional neural networks. The proposed approach employs RGB-D image data acquired from an eye-in-hand camera centering the object of interest in the field of view using visual servoing. Our approach aims at reducing propagation errors and eliminating the need for complex hand tracking algorithm, image segmentation, or 3D reconstruction. The proposed approach is able to efficiently generate reliable multi-view object grasps regardless of the geometric complexity and physical properties of the object in question. The proposed system architecture enables simple and effective path generation and a real-time tracking control. In addition, our system is modular, reliable, and accurate in both end effector path generation and control. We experimentally justify the efficacy and effectiveness of our overall system on the Barrett Whole Arm Manipulator.

PackerRobo: Model-based robot vision self supervised learning in CART

Robots are most widely used to replace human contribution with machine generated response. When humans interact with robots, its mandatory for both to forecast actions based on current conditions. Huge efforts have been channelized towards attaining this perfect coordination. To decipher complex environments, the inference of robotic mobility and alteration of random unstructured scenarios is a complicated task in the field of visual processing and imaging. To address this issue, a new Vision-Based Interaction Model based on deep neural networks has been suggested. The proposed model solves the error amplification issue by the application of past inputs through features as reposed by a Deep Belief Network (DBN). In addition, a novel Vision-Based Robotics Learning model is also proposed for scene understanding and recognition using deep neural network understanding. Moreover, a vision theory-based smart learning algorithm is also suggested to decide positive possible outcomes.Therefore, the model is capable of using object motions to extract relevant information used for Turning, Griping and object mobility.To validate the suggested model, a number of experiments have been performed on benchmark datasets and it showed a higher performance as evaluated against some of the niche methods.

General Geometry Calibration Using Arbitrary Free-Form Surface in a Vision-Based Robot System

Geometry calibration is a critical problem in vision-based robot systems. The calibration objects of existing methods are limited to regular shapes. In this article, a general calibration method using an arbitrary free-form surface is proposed to simultaneously calibrate the geometry parameters. By incorporating a shape matching algorithm, each measured point on the surface can be regarded as a feature point to compare with the design model for a closed-form initial solution and an iterative fine solution. In the objective function of fine solution, the residual is described by the point-to-tangent distance, and the solution is proved to be Gaussian-Newton method with second-order convergence. The geometry and matching errors are iteratively compensated to improve the calibration accuracy. The characteristics of the method are large number of feature points, no need for specific features, no limitations on the size of the free-form surface and convenient robot pose control. Finally, simulations and experiments verify the availability of the proposed method in the presence of measuring noise, robot repeated positioning error, and a small number of robot poses.