Vision-based Robot Research Articles

Exponentially stable observer-based controller for VTOL-UAVs without velocity measurements

There is a great demand for vision-based robotics solutions that can operate using Global Positioning Systems (GPS), but are also robust against GPS signal loss and gyroscope failure. This paper investigates the estimation and tracking control in application to a Vertical Take-Off and Landing (VTOL) Unmanned Aerial Vehicle (UAV) in six degrees of freedom (6 DoF). A full state observer for the estimation of VTOL-UAV motion parameters (attitude, angular velocity, position, and linear velocity) is proposed on the Lie Group of with almost globally exponentially stable closed-loop error signals. Thereafter, a full state observer-based controller for the VTOL-UAV motion parameters is proposed on the Lie Group with a guaranteed almost global exponential stability. The proposed approach produces good results without the need for angular and linear velocity measurements (without a gyroscope and GPS signals) utilising only a set of known landmarks obtained by a vision-aided unit (monocular or stereo camera). The equivalent quaternion representation on is provided in the Appendix. The observer-based controller is presented in a continuous form while its discrete version is tested using a VTOL-UAV simulation that incorporates large initial error and uncertain measurements. The proposed observer is additionally tested experimentally on a real-world UAV flight dataset.

A Vision-Based Robot Grasping System

Grasping is critical for intelligent robots to accomplish sophisticated tasks. Even with multimodal sensor fusion, accurately and reliably estimating grasp poses for complex-shaped objects remains a challenge. In this paper, we design a vision-based grasping platform for a more general case, that is, grasping a variety of objects by a simple parallel gripper with the grasp detection model consuming RGB sensing or depth sensing. Focusing on the grasp pose estimation part, we propose a deep grasp detector that uses a densely connected Feature Pyramid Network (FPN) feature extractor and multiple two-stage detection units to achieve dense grasp pose predictions. Specifically, for the feature extractor, the fusion of different layer feature maps can increase both the model’s capacity to detect the various size grasp areas and the accuracy of the regressed grasp positions. For each of the two-stage detection unit, the first stage generates horizontal candidate grasp areas, while the second stage refines them to predict the rotated grasp poses. We train and validate our grasp pose estimation algorithm on the Cornell Grasp Dataset and the Jacquard Dataset. The model achieves the detection accuracy of 93.3% and 89.6%, respectively. We further design real-world grasp experiments to verify the effectiveness of our vision-based robotic grasping system. The real scenario trials validate that the system is capable of grasping unseen objects, in particular, achieving robust and accurate grasp pose detection and gripper opening width measurement based on depth sensing only.

Vision-Only Robot Navigation in a Neural Radiance World

Neural Radiance Fields (NeRFs) have recently emerged as a powerful paradigm for the representation of natural, complex 3D scenes. Neural Radiance Fields (NeRFs) represent continuous volumetric density and RGB values in a neural network, and generate photo-realistic images from unseen camera viewpoints through ray tracing. We propose an algorithm for navigating a robot through a 3D environment represented as a NeRF using only an onboard RGB camera for localization. We assume the NeRF for the scene has been pre-trained offline, and the robot’s objective is to navigate through unoccupied space in the NeRF to reach a goal pose. We introduce a trajectory optimization algorithm that avoids collisions with high-density regions in the NeRF based on a discrete time version of differential flatness that is amenable to constraining the robot’s full pose and control inputs. We also introduce an optimization based filtering method to estimate 6DoF pose and velocities for the robot in the NeRF given only an onboard RGB camera. We combine the trajectory planner with the pose filter in an online replanning loop to give a vision-based robot navigation pipeline. We present simulation results with a quadrotor robot navigating through a jungle gym environment, the inside of a church, and Stonehenge using only an RGB camera. We also demonstrate an omnidirectional ground robot navigating through the church, requiring it to reorient to fit through a narrow gap.

Multi-Stage Refinement Feature Matching Using Adaptive ORB Features for Robotic Vision Navigation

Feature matching is a crucial technique to estimate robot pose in the light of natural landmarks for a vision navigation system. It is difficult for the present feature matching approaches to balance the accuracy, robustness and efficiency for a vision-based robot in complex workspaces. A multi-stage KNN-TF-BF-EN-RANSAC (KTBER) refinement matching technique is proposed to combine with the adaptive ORB feature extraction method, in order to develop a novel KTBER_AORB model that can outperform the state-of-the-art feature extraction and matching techniques. Firstly, the adaptive ORB (AORB) features are extracted in the Laplacian Transformation of Gaussian (LTOG) pyramid by using a set of variable-radius templates. The template radius is adjusted in terms of different brightness and contrast conditions. Then, the Eigenvector Norm (EN) and Bidirectional Filtering (BF) techniques are devised to establish a novel KTBER model, which can match AORB feature pairs not only according to the local appearance-based similarity of feature descriptors in the Hamming space, but also by means of the similarity of local image structure and global grayscale information for feature pairs in the Euclidean space. Finally, a large number of feature matching experiments are conducted on five public datasets and our mobile robot dataset, in order to test the algorithm performances for different challenging scenarios, including illumination change, motion blur, image rotation, viewpoint change, and low texture. On the one hand, an intensive experimental analysis of our model with different configurations is made to check the specific effects of different algorithm elements. On the other hand, the KTBER_AORB model is further compared with four state-of-the-art techniques, e.g., SIFT, ORB-SLAM, MP-ORB, and LAAT. The experimental results demonstrate that the KTBER_AORB model outperforms other state-of-the-art feature matching techniques on accuracy and robustness, still preserving a higher efficiency than most counterparts.

A Fully Automatic Calibration for Vision-Based Selective Compliance Assembly Robot Arm and Its Application to Intelligent Wafer Inspection Scheduling

A Fully Automatic Calibration for Vision-Based Selective Compliance Assembly Robot Arm and Its Application to Intelligent Wafer Inspection Scheduling

Robust Pose Estimation via Hybrid Point and Twin Line Reprojection for RGB-D Vision Navigation

Pose estimation is a crucial technique to achieve robot localization using natural features for an RGB-D vision navigation system. However, it still suffers from the impoverishment of robustness and accuracy in complex workspaces. A robust real-time pose estimation approach using hybrid point and twin line reprojection is proposed to allow accurate localization for a vision-based robot under various challenging scenarios including low-texture, low-structure, occlusion, and viewpoint and illumination changes. Firstly, an improved adaptive threshold (IAT)-based oriented fast and rotated Brief (ORB) feature detection method is developed to extract sufficient IAT-ORB point features, which are matched by a multi-stage KNN-RTC-CS-RANSAC (KRCR) refinement feature matching strategy to generate reliable point matches. Then a robust EDLines line feature matching method is developed by integrating dynamic matching (DM) and median absolute deviation (MAD) to obtain fine line matches. Furthermore, the hybrid point reprojection errors are built from the fusion of 3D-2D and 3D-3D point pairs, and the twin line reprojection errors are constructed by integrating virtual right-eye lines generated by depth measurement, which both are incorporated into a novel unified errors optimization model to achieve optimal motion estimation by minimizing the reprojection errors of points and lines concurrently. Finally, the extensive experiments verify the robustness, accuracy, and real-time performance of the proposed approach on the public datasets and our real indoor scenarios. The quantitative results demonstrate that our proposed approach gains more accurate and stable localization than other state-of-the-art methods under challenging environments.

An Adaptive Pose Fusion Method for Indoor Map Construction

The vision-based robot pose estimation and mapping system has the disadvantage of low pose estimation accuracy and poor local detail mapping effects, while the modeling environment has poor features, high dynamics, weak light, and multiple shadows, among others. To address these issues, we propose an adaptive pose fusion (APF) method to fuse the robot’s pose and use the optimized pose to construct an indoor map. Firstly, the proposed method calculates the robot’s pose by the camera and inertial measurement unit (IMU), respectively. Then, the pose fusion method is adaptively selected according to the motion state of the robot. When the robot is in a static state, the proposed method directly uses the extended Kalman filter (EKF) method to fuse camera and IMU data. When the robot is in a motive state, the weighted coefficient is determined according to the matching success rate of the feature points, and the weighted pose fusion (WPF) method is used to fuse camera and IMU data. According to the different states, a series of new poses of the robot are obtained. Secondly, the fusion optimized pose is used to correct the distance and azimuth angle of the laser points obtained by LiDAR, and a Gauss–Newton iterative matching process is used to match the corresponding laser points to construct an indoor map. Finally, a pose fusion experiment is designed, and the EuRoc data and the measured data are used to verify the effectiveness of this method. The experimental results confirm that this method provides higher pose estimation accuracy compared with the robust visual inertial odometry (ROVIO) and visual-inertial ORB-SLAM (VI ORB-SLAM) algorithms. Compared with the Cartographer algorithm, this method provides higher two-dimensional map modeling accuracy and modeling performance.

3D Recognition Based on Sensor Modalities for Robotic Systems: A Survey.

3D visual recognition is a prerequisite for most autonomous robotic systems operating in the real world. It empowers robots to perform a variety of tasks, such as tracking, understanding the environment, and human–robot interaction. Autonomous robots equipped with 3D recognition capability can better perform their social roles through supportive task assistance in professional jobs and effective domestic services. For active assistance, social robots must recognize their surroundings, including objects and places to perform the task more efficiently. This article first highlights the value-centric role of social robots in society by presenting recently developed robots and describes their main features. Instigated by the recognition capability of social robots, we present the analysis of data representation methods based on sensor modalities for 3D object and place recognition using deep learning models. In this direction, we delineate the research gaps that need to be addressed, summarize 3D recognition datasets, and present performance comparisons. Finally, a discussion of future research directions concludes the article. This survey is intended to show how recent developments in 3D visual recognition based on sensor modalities using deep-learning-based approaches can lay the groundwork to inspire further research and serves as a guide to those who are interested in vision-based robotics applications.

Review: Teknologi Simple Phenotyping sebagai Database Pengembangan Robot Pendeteksi dan Pemupuk Nitrogen Padi

Rice is one of the most important staple food consumed by more than half of the world population, especially in Indonesia where majority of the people consumed rice as daily intake. Precision agriculture is needed to boost production to meet the demand. As limiting factor for plant growth , huge doses of nitrogen fertilizer are deployed, in fact the absorbtion rate of plant was only about 50%. Developing vision-control based robot for fertilizing plant need proper database so that false positive and negative c ould be avoided. Several previous methods that were available to assest nitrogen status in plant such as a color chart, spectrometer, SPAD-502, and kjedah l . Plant phenotyping is often used for plant breeding, but also very useful in determining nutrient status in plant s . Cheap and simple phenotyping using digital camera followed by extracting data through open source platform such as ImageJ can be very useful to generate data for building the database. Here we presented the current progress on plant phenotyping for nitrogen assessment in plant as well as an ImageJ in phenomic era. We also highlight some robotic progress especially vision-based robot, who rely on proper imaging data for their training .

Vision-based robot identification and tracking

In recent years, computer vision technology in the field of science and technology has been stubbornly high, making a lot of scientific and technical personnel involved in the research gate technology. Depth learning, artificial intelligence in recent years, often around our ears, vision technology is occupying a place in the field of artificial intelligence. This paper mainly studies the recognition and localization of robots based on vision. The mobile robot is detected by the background difference method, and the background model is established by the Gaussian background modeling. Recognizing the color and number tags on the robot for matching the robot’s features. Real-time positioning of the robot under the premise of the same background.

Design and Simulation of a Vision-Based Automatic Trout Fish-Processing Robot

Today, industrial automation is being applied in a wide range of fields. The initial modeling of robots and mechanical systems together with simulation results in optimal systems. In this study, the designed system is simulated to obtain the required velocities, accelerations and torques of the actuating arms in a vision-based automatic system. Due to the slippery skin of fish and the low friction coefficient, it is not easy to design an optimal tool to handle fish. Since the fish-processing operation is undertaken step by step and provides fish stability, it is essential that the gripper enables different processing operations along the system. The proposed system performs belly-cutting, beheading, gutting, and cleaning stages for different fish sizes, based on the extracted dimensions of the vision system. In the head-cutting section, the average speed of the actuator jack was considered as 500 mm s−1. Under these conditions, the maximum required force to provide this speed was 332.45 N. In the belly-cutting subsystem, the required torque for the stepper motor resulted in 1.79–2.15 N m. Finally, the maximum required torque for the gutting stepper motor was calculated as 0.69 N m in the tested processing capacities.

Research on Adaptive Grasping of Robotic Manipulator with Depth Visual Perception

Aiming at the problem of poor grasping performance and low repetitive teaching efficiency when traditional machine vision-based robots are facing the grasping tasks of objects with unknown heights, an adaptive grasping method of robotic manipulator based on depth visual perception feedback control is proposed. The RGB-D image is acquired through the depth camera, and the embedded computing device is used to process the RGB-D image based on ROS(Robot Operating System) to obtain the object positioning information. Finally, the coordinate information of the object is used to control the joint motion of the robot to perform the grasping task. The experimental results show that the method is robust and requires a small amount of data, and it can grasp the target object with unknown height without teaching.

Real-Time Object Detection and Localization for Vision-Based Robot Manipulator

Traditionally, the robot manipulators were assigned a monotonous task. Therefore, in an aim of simplifying the interactions between unstructured surrounding and robots, as well as to carry out the more complex operations, cognitive abilities such as visual perception are appended to systems for the sake of intelligent operations. Hence, this article directed towards the multistage process involved in the design and development of vision-based robot manipulator suitable for pick and place operation in real-time entailing external disturbances. The intact system development occurs in three phases: first, the robot manipulator with 3-DOF is designed, developed and also enhanced by integrating the vision source. Second, an algorithm for object detection and localization in real time is framed in which segmented object matrices are returned by the former and the position of the stabilized object is returned by the latter. Third, entire hardware–software integration is achieved to perform the desired operation. The algorithm developed in this paper proves to be good for the developed vision-based manipulator, as we achieve quite a good accuracy in object detection algorithm whereas 98.41–99.12% accuracy is achieved in the object localization algorithm.

Outline viewpoint feature histogram: An improved point cloud descriptor for recognition and grasping of workpieces.

We propose an improved point cloud global descriptor for recognition and grasping of similar workpieces. In the industry, different types of workpieces need to be recognized precisely in some intelligent systems. Deep learning requires a lot of preparation work, and it is difficult to adapt to the variety of workpieces. Furthermore, traditional descriptors based on point pairs cannot meet the requirements of identification. To solve this problem, the Outline Viewpoint Feature Histogram (Outline-VFH) descriptor remains part of the recognition ability of the Viewpoint Feature Histogram (VFH) descriptor and contains an extra outline description, which is established based on the oriented bounding box theory. To validate the effectiveness of the proposed descriptor, experiments were conducted on public dataset and some physical workpieces. The results show that the Outline-VFH is much better than VFH and some other descriptors on recognition and has great potential in vision-based robot grasping applications.

Deep Learning applications for COVID-19

This survey explores how Deep Learning has battled the COVID-19 pandemic and provides directions for future research on COVID-19. We cover Deep Learning applications in Natural Language Processing, Computer Vision, Life Sciences, and Epidemiology. We describe how each of these applications vary with the availability of big data and how learning tasks are constructed. We begin by evaluating the current state of Deep Learning and conclude with key limitations of Deep Learning for COVID-19 applications. These limitations include Interpretability, Generalization Metrics, Learning from Limited Labeled Data, and Data Privacy. Natural Language Processing applications include mining COVID-19 research for Information Retrieval and Question Answering, as well as Misinformation Detection, and Public Sentiment Analysis. Computer Vision applications cover Medical Image Analysis, Ambient Intelligence, and Vision-based Robotics. Within Life Sciences, our survey looks at how Deep Learning can be applied to Precision Diagnostics, Protein Structure Prediction, and Drug Repurposing. Deep Learning has additionally been utilized in Spread Forecasting for Epidemiology. Our literature review has found many examples of Deep Learning systems to fight COVID-19. We hope that this survey will help accelerate the use of Deep Learning for COVID-19 research.

Calibration-Free Monocular Vision-Based Robot Manipulations With Occlusion Awareness

Vision-based manipulation has been largely used in various robot applications. Normally, in order to obtain the spatial information of the operated target, a carefully calibrated stereo vision system is required. However, it limits the application of robots in the unstructured environment which limits both the number and the pose of the camera. In this study, a calibration-free monocular vision-based robot manipulation approach is proposed based on domain randomization and deep reinforcement learning (DRL). Firstly, a learning strategy combined domain randomization is developed to estimate the spatial information of the target from a single monocular camera arbitrarily mounted in a large area of the manipulation environment. Secondly, to address the monocular occlusion problem which regularly happens during robot manipulations, an occlusion awareness DRL policy has been designed to control the robot to avoid occlusions actively in the manipulation tasks. The performance of our method has been evaluated on two common manipulation tasks, reaching and lifting of a target building block, which show the efficiency and effectiveness of our proposed approach.

Robotic Arm Assistance System Based on Simple Stereo Matching and Q-Learning Optimization

This study presents a stereo vision robotic arm assistance system, in which five degrees of catching can be performed by the robot arm in a single instance. The algorithm of the control system is built for population-based optimization and specifically aimed to assist people with disabilities. The proposed stereo vision-based robot arm system enables users to manipulate objects based on the robot’s ability to aim at objects by using computer vision. The stereo vision system counts the parameters by focusing on the real-world position of the instance in the coordinate system. A trained deep fully connected network is then adopted to compensate the location measurement errors incurred by the inaccurate parameters measured from the deep learning procedure. Subsequently, the proposed Q-learning-based swarm optimization algorithm is adopted to solve the forward kinematics problem and count the angles of each servo. The performance of the robot arm is compared with several real-life experiments to test its ability to grip a target object in different positions.

Proactive Camera Attribute Control Using Bayesian Optimization for Illumination-Resilient Visual Navigation

Illumination variance is a major challenge for vision-based robotics. Most approaches focus on alleviating illumination changes in already captured images. Despite the large utility, camera attributes have been empirically determined to function in a highly passive manner, yielding vision algorithm failure under radical illumination variance. Recent studies have proposed exposure and gain control schemes that could maximize image information and eschew saturation. In this article, we propose a proactive control scheme for the camera's two dominant attributes—exposure time and gain control. Unlike existing approaches, we formulate this camera attribute control as an optimization problem in which the underlying function is not known a priori . We first define a new metric of the image regarding these two major attributes to include both image gradients and signal-to-noise ratio simultaneously. Based on this metric, we introduce a new formulation for this attribute control via Bayesian optimization (BO) and learn the environmental change from the captured image. During the control, to mitigate the burden of image acquisition and Bayesian optimization, images are synthesized using a camera response function and avoided the actual frame grab from the camera. The proposed method was validated in light-flickering indoor, outdoor near sunset, and indoor–outdoor transient environments where light changes rapidly, supporting 20–40 Hz frame rates.

Path Generation and Map Construction for Mobile Robot Navigation

Vision based mobile robot navigation is one of the important because the robots can resolve lots of problems in which human life is dangerous, difficult and costly. A difficult problem is to make a vision-based robot navigate properly to the target by avoiding the obstacles. Path information is very essential for successful navigation of the robot. The main purpose of this paper is to present an approach for mobile robots to generate path and construct maps. This work is focused on providing the path information of an environment using a single 2-D camera. Whenever robot identifies the human in the corridor, it generates the information of the path information and constructs the map is achieved based on the human moment and the human position. On the robot the camera is mounted which acquires images in real time. Oath information is generated for mobile robot using acquired real time images and processing real time images.

A fast calibration method of the robot’s 3D vision system

Calibration accuracy and convenience are very important for the rapid production of vision-based robots. Although there are many calibration methods for the 3D vision system, the convenience of calibration cannot meet the needs of industrial sites. In this paper, we propose a fast calibration method based on a 3D calibration block. Its key point is to complete the calibration process through multiple space point constraints, and the space points used are jointly solved by multiple planes obtained by local point cloud fitting. The experimental results show that the proposed method can quickly complete the calibration of the robot’s 3D vision system, and the calibration accuracy can meet low precision occasions.