Robot Vision Applications Research Articles

Shadow Detection and Elimination for Robot and Machine Vision Applications

Shadow removal is crucial for robot and machine vision as the accuracy of object detection is greatly influenced by the uncertainty and ambiguity of the visual scene. In this paper, we introduce a new algorithm for shadow detection and removal based on different shapes, orientations, and spatial extents of Gaussian equations. Here, the contrast information of the visual scene is utilized for shadow detection and removal through five consecutive processing stages. In the first stage, contrast filtering is performed to obtain the contrast information of the image. The second stage involves a normalization process that suppresses noise and generates a balanced intensity at a specific position compared to the neighboring intensities. In the third stage, the boundary of the target object is extracted, and in the fourth and fifth stages, respectively, the region of interest (ROI) is highlighted and reconstructed. Our model was tested and evaluated using realistic scenarios which include outdoor and indoor scenes. The results reflect the ability of our approach to detect and remove shadows and reconstruct a shadow free image with a small error of approximately 6%.

Visual Odometry of a Low-Profile Pallet Robot Based on Ortho-Rectified Ground Plane Image from Fisheye Camera

This study presents a visual-only odometry technique of a low-profile pallet robot using image feature tracking in ground plane images generated from a fisheye camera. The fisheye camera is commonly used in many robot vision applications because it provides a larger field of view (FoV) around a robot. However, because of the large radial distortion, the fisheye image is generally converted to a pinhole image for visual feature tracking or matching. Although the radial distortion can be eliminated via image undistortion with the lens calibration parameters, it causes several side effects, such as degraded image resolution and a significant reduction in the FoV. In this paper, instead of using the pinhole model, we propose to generate a ground plane image (GPI) from the fisheye image. GPI is a virtual top-view image that only contains the ground plane at the front of the robot. First, the original fisheye image is projected to several virtual pinhole images to generate a cubemap. Second, the front and bottom faces of the cubemap are projected to a GPI. Third, the GPI is homographically transformed again to further reduce image distortion. As a result, an accurate ortho-rectified ground plane image is obtained from the virtual top-view camera. For visual odometry using the ortho-rectified GPI, a number of 2D motion vectors are obtained using feature extraction and tracking between the previous and current frames in the GPI. By calculating a scaled motion vector, which is the measurement of the virtual wheel encoder of the mobile robot, we estimate the velocity and steering angle of the virtual wheel using the motion vector. Finally, we estimate the pose of the mobile robot by applying a kinematic model to the mobile robot.

Evaluation of Single and Dual image Object Detection through Image Segmentation Using ResNet18 in Robotic Vision Applications

This study presents a method for enhancing the accuracy of object detection in industrial automation applications using ResNet18-based image segmentation. The objective is to extract object images from the background image accurately and efficiently. The study includes three experiments, RGB to grayscale conversion, single image processing, and dual image processing. The results of the experiments show that dual image processing is superior to both RGB to grayscale conversion and single image processing techniques in accurately identifying object edges, determining CG values, and cutting background images and gripper heads. The program achieved a 100% success rate for objects located in the workpiece tray, while also identifying the color and shape of the object using ResNet-18. However, single image processing may have advantages in certain scenarios with sufficient image information and favorable lighting conditions. Both methods have limitations, and future research could focus on further improvements and optimization of these methods, including separating objects into boxes of each type and converting image coordinate data into robot working area coordinates. Overall, this study provides valuable insights into the strengths and limitations of different object recognition techniques for industrial automation applications.

A Normalized Disparity Loss for Stereo Matching Networks

Stereo vision is an essential component for lots of robot vision applications. The performance of stereo matching has been greatly improved recently thanks to deep learning technologies. However, most of the current deep networks directly use naive disparity loss to train the network, regardless of the inherent unbalance of its supervision to pixels with different depths. In this letter, we propose a normalized disparity loss to solve this problem. The new loss is constructed as the common disparity loss divided by a fitted cost function. This function is responsible for imitating the variance of the common loss along the disparity dimension. Range restriction and gradual updating strategies are further designed to ensure the robustness of the loss function during training. Experimental results on multiple public datasets show that our method can be directly applied to most state-of-the-art networks and effectively improve their disparity performance.

Metasurface-driven full-space structured light for three-dimensional imaging

Structured light (SL)-based depth-sensing technology illuminates the objects with an array of dots, and backscattered light is monitored to extract three-dimensional information. Conventionally, diffractive optical elements have been used to form laser dot array, however, the field-of-view (FOV) and diffraction efficiency are limited due to their micron-scale pixel size. Here, we propose a metasurface-enhanced SL-based depth-sensing platform that scatters high-density ~10 K dot array over the 180° FOV by manipulating light at subwavelength-scale. As a proof-of-concept, we place face masks one on the beam axis and the other 50° apart from axis within distance of 1 m and estimate the depth information using a stereo matching algorithm. Furthermore, we demonstrate the replication of the metasurface using the nanoparticle-embedded-resin (nano-PER) imprinting method which enables high-throughput manufacturing of the metasurfaces on any arbitrary substrates. Such a full-space diffractive metasurface may afford ultra-compact depth perception platform for face recognition and automotive robot vision applications.

Uncertainty Guided Policy for Active Robotic 3D Reconstruction Using Neural Radiance Fields

In this letter, we tackle the problem of active robotic 3D reconstruction of an object. In particular, we study how a mobile robot with an arm-held camera can select a favorable number of views to recover an object's 3D shape efficiently. Contrary to the existing solution to this problem, we leverage the popular neural radiance fields-based object representation, which has recently shown impressive results for various computer vision tasks. However, it is not straightforward to directly reason about an object's explicit 3D geometric details using such a representation, making the next-best-view selection problem for dense 3D reconstruction challenging. This paper introduces a ray-based volumetric uncertainty estimator, which computes the entropy of the weight distribution of the color samples along each ray of the object's implicit neural representation. We show that it is possible to infer the uncertainty of the underlying 3D geometry given a novel view with the proposed estimator. We then present a next-best-view selection policy guided by the ray-based volumetric uncertainty in neural radiance fields-based representations. Encouraging experimental results on synthetic and real-world data suggest that the approach presented in this paper can enable a new research direction of using an implicit 3D object representation for the next-best-view problem in robot vision applications, distinguishing our approach from the existing approaches that rely on explicit 3D geometric modeling.

Development of an autonomous fog computing platform using control-theoretic approach for robot-vision applications

This paper presents the dynamic modelling and linear matrix inequality (LMI) based controller design of a distributed fog computing framework for real-time robot vision applications. A mobile robot vision system acquires the images from an application environment such as a warehouse, where articles are stacked in numerous racks. We characterise the mobile robot vision data (MRVD) using frames per second (FPS) and the image resolution. From the MRVD, object detection is performed by an open-source deep learning (DL) platform for detecting and localising various objects. However, with higher FPS together with high-resolution images, the processing time by the DL algorithm increases significantly. This necessitates the deployment of a distributed computing platform with several computing nodes. In this work, we deploy a distributed fog computing environment (DFCE) for the real-time object detection in an application environment.The processing time required to handle the MRVD is called the service time. However, for efficient auto-scaling performance, the mathematical model of the DFCE, taking into consideration the characteristics of the MRVD is necessary. In this context, we envisage the application of control theory to build the dynamic model of the DFCE. A Linear Parameter Varying (LPV) model is proposed for the DFCE with the service time as the output, the number of fog nodes as the input, and the characteristics of MRVD as the time-varying parameters. At first, an LPV model for the DFCE is derived using system identification, and the model is validated using the real-time test data. The LPV model is converted to a Polytopic LPV (PLPV) model for LMI based controller design. Finally, we develop and validate a Linear Matrix Inequality (LMI) based LPV controller to meet the service time constraints for a given application environment.For localisation and trajectory tracking with obstacle avoidance in the application environment, the mobile robot implements an Extended Kalman Filter (EKF) based simultaneous localisation and mapping (SLAM) and a bug-based path planning algorithm respectively. Finally, we present, detailed controller validation results illustrating the mobile robot navigation together with the auto-scaling control for the fog computing platform to modulate the service time.

Deep Learning-Based Robot Vision: High-End Tools for Smart Manufacturing

With the development of smart manufacturing technology, manufacturers all over the world have begun using automated processing and unmanned factories. China already has many high-end production lines that use intelligent robots, where Artificial Intelligence (AI) and robot vision play a critical role in facilitating robot-driven production. This paper provides a review of the state-of-the-art in Deep Learning (DL) techniques and the applications of DL-based robot vision in smart manufacturing, with an emphasis on the contributions made by China. We conclude by summarizing challenges and prospects in the area.

Perspectives of RealSense and ZED Depth Sensors for Robotic Vision Applications

This review paper presents an overview of depth cameras. Our goal is to describe the features and capabilities of the introduced depth sensors in order to determine their possibilities in robotic applications, focusing on objects that might appear in applications with high accuracy requirements. A series of experiments was conducted, and various depth measuring conditions were examined in order to compare the measurement results of all the depth cameras. Based on the results, all the examined depth sensors were appropriate for applications where obstacle avoidance and robot spatial orientation were required in coexistence with image vision algorithms. In robotic vision applications where high accuracy and precision were obligatory, the ZED depth sensors achieved better measurement results.

Deploying Machine Learning Techniques for Human Emotion Detection.

Emotion recognition is one of the trending research fields. It is involved in several applications. Its most interesting applications include robotic vision and interactive robotic communication. Human emotions can be detected using both speech and visual modalities. Facial expressions can be considered as ideal means for detecting the persons' emotions. This paper presents a real-time approach for implementing emotion detection and deploying it in the robotic vision applications. The proposed approach consists of four phases: preprocessing, key point generation, key point selection and angular encoding, and classification. The main idea is to generate key points using MediaPipe face mesh algorithm, which is based on real-time deep learning. In addition, the generated key points are encoded using a sequence of carefully designed mesh generator and angular encoding modules. Furthermore, feature decomposition is performed using Principal Component Analysis (PCA). This phase is deployed to enhance the accuracy of emotion detection. Finally, the decomposed features are enrolled into a Machine Learning (ML) technique that depends on a Support Vector Machine (SVM), k-Nearest Neighbor (KNN), Naïve Bayes (NB), Logistic Regression (LR), or Random Forest (RF) classifier. Moreover, we deploy a Multilayer Perceptron (MLP) as an efficient deep neural network technique. The presented techniques are evaluated on different datasets with different evaluation metrics. The simulation results reveal that they achieve a superior performance with a human emotion detection accuracy of 97%, which ensures superiority among the efforts in this field.

Formulating Event-Based Image Reconstruction as a Linear Inverse Problem With Deep Regularization Using Optical Flow.

Event cameras are novel bio-inspired sensors that measure per-pixel brightness differences asynchronously. Recovering brightness from events is appealing since the reconstructed images inherit the high dynamic range (HDR) and high-speed properties of events; hence they can be used in many robotic vision applications and to generate slow-motion HDR videos. However, state-of-the-art methods tackle this problem by training an event-to-image Recurrent Neural Network (RNN), which lacks explainability and is difficult to tune. In this work we show, for the first time, how tackling the combined problem of motion and brightness estimation leads us to formulate event-based image reconstruction as a linear inverse problem that can be solved without training an image reconstruction RNN. Instead, classical and learning-based regularizers are used to solve the problem and remove artifacts from the reconstructed images. The experiments show that the proposed approach generates images with visual quality on par with state-of-the-art methods despite only using data from a short time interval. State-of-the-art results are achieved using an image denoising Convolutional Neural Network (CNN) as the regularization function. The proposed regularized formulation and solvers have a unifying character because they can be applied also to reconstruct brightness from the second derivative. Additionally, the formulation is attractive because it can be naturally combined with super-resolution, motion-segmentation and color demosaicing. Code is available at https://github.com/tub-rip/event_based_image_rec_inverse_problem.

Nonlinear dimensionality reduction in robot vision for industrial monitoring process via deep three dimensional Spearman correlation analysis (D3D-SCA)

During the era of Industry 4.0, the industrial robot monitoring process is getting success and popularity day by day. It also plays a vital role in the enhancement of robot vision algorithms. This paper proposed a model Deep Three Dimensional Spearman Correlation Analysis (D3D-SCA) to address nonlinear dimensionality reduction in robot vision for three-dimensional data. Dealing with three-dimensional multimedia datasets using traditional algorithms, to date, researchers have been facing limitations and challenges because mostly sub-space learning algorithms and their developments cannot perform satisfactorily in most of the time with linear and non-linear data dependency. The proposed model directly finds the relations between two sets of three-dimensional data without reshaping the data into 2D-matrices or vectors and dramatically reduces the dimensional reduction and computational algorithm complexity. The proposed model extracts deep information and translates it into a decision. To do so, three components are employed in the proposed model: customized deep learning model Inception_V3 for deep feature mapping, three-dimensional spearman correlation analysis for comparing pairwise deep features without a singular matrix and spatial dilemma problem, and the customized Xception classifier with automatic online updating ability and adjustable neural architecture for low latency models. The motivation of the proposed model is to advance the scalability of existing industrial robot vision applications which based on recognition, detection and re-identification approaches. Extensive findings on industrial datasets named “3D Objects on turntable and Caltech 101” demonstrate the effectiveness of the proposed model.

Trajectory Drift–Compensated Solution of a Stereo RGB-D Mapping System

Multiple sensors are commonly used for three-dimensional (3D)-mapping or robotic-vision applications, as they provide a larger field of view and sufficient observations to fulfill frame-registration and map-updating tasks. However, the data sequences generated by multiple sensors can be inconsistent and contain significant time drift. In this paper, we describe the trajectory drift–compensated strategy that we designed to eliminate the influence of time drift between sensors, remove the inconsistency between the sequences from various sensors, and thereby generate a coarse-to-fine procedure for robust camera-tracking based on two-dimensional–3D observations from stereo sensors. We present the mathematical analysis of the iterative optimizations for pose tracking in a stereo red, green, blue plus depth (RGB-D) camera. Finally, complex indoor scenario experiments demonstrate the efficiency of the proposed stereo RGB-D simultaneous localization and mapping solution. The results verify that the proposed stereo RGB-D mapping solution effectively improves the accuracies of both camera-tracking and 3D reconstruction.

Robot vision application on embedded vision implementation with digital signal processor

The great development of robot vision represented by deep learning places urgent demands on embedded vision implementation. This article introduces a hardware framework for implementation of embedded vision based on digital signal processor, which can be widely used in robot vision applications. Firstly, the article discusses implementation of a pretrained typical convolutional neural network on the digital signal processor embedded system for real-time handwritten digit recognition. Then, the article introduces the migration of OpenCV software packages to digital signal processor embedded system and the implementation flow of face detection algorithms with OpenCV on digital signal processor. The experimental results are remarkable with convolutional neural networks for handwritten digit recognition. This article provides a convenient and feasible design scheme of digital signal processor system for implementation of embedded vision.

A simulated annealing algorithm to quantify patterns in astronomical data

ABSTRACT We develop an optimization algorithm, using simulated annealing for the quantification of patterns in astronomical data based on techniques developed for robotic vision applications. The methodology falls in the category of cost minimization algorithms and it is based on user-determined interaction – among the pattern elements – criteria that define the properties of the sought structures. We applied the algorithm on a large variety of mock images and we constrained the free parameters; α and k, which express the amount of noise in the image and how strictly the algorithm seeks for cocircular structures, respectively. We find that the two parameters are interrelated and also that, independently of the pattern properties, an appropriate selection for most of the images would be log k = −2 and 0 < α ≲ 0.04. The width of the effective α-range, for different values of k, is reduced when more interaction coefficients are taken into account for the definition of the patterns of interest. Finally, we applied the algorithm on N-body simulation dark-matter halo data and on the HST image of the lensing Abell 2218 cluster to conclude that this versatile technique could be applied for the quantification of structure and for identifying coherence in astronomical patterns.

Solving the depth ambiguity in single-perspective images

Scene depth estimation is gaining importance as more and more AR/VR and robot vision applications are developed. Conventional depth-from-defocus techniques can passively provide depth maps from a single image. This is especially advantageous for moving scenes. However, they suffer a depth ambiguity problem where two distinct depth planes can have the same amount of defocus blur in the captured image. We solve the ambiguity problem and, as a consequence, introduce a passive technique that provides a one-to-one mapping between depth and defocus blur. Our method relies on the fact that the relationship between defocus blur and depth is also wavelength dependent. The depth ambiguity is thus solved by leveraging (multi-) spectral information. Specifically, we analyze the difference in defocus blur of two channels to obtain different scene depth regions. This paper provides the derivation of our solution, a robustness analysis, and validation on consumer lenses.

Online Semantic Object Segmentation for Vision Robot Collected Video

Video segmentation is the foundation of many senior computer vision applications. For robotic vision applications, video object segmentation is facing more difficulties. Since the videos are usually collected from those complicated scenes and the camera perspective may be static for some time, object saliency and motion clues which are widely used in traditional video segmentation approaches could be unreliable. Besides, its online processing ability is of crucial importance. In this paper, we propose an online video object segmentation method, which is organized as a segmentation-by-detection framework. It first effectively segments out meaningful objects in short video frame batches with the help of object detectors, and then the segmentation of different batches can be associated with each other via bi-directional notebook based connecting strategy. Experiments conducted on public available datasets verify the above-mentioned property and show good performance of the proposed method.

A Method of Camera Link Image Data Processing Based on FPGA

Camera Link is based on the development of Channel Link Technology. The data is transmitted using low voltage differential signal (LVDS). It has some advantages, such as fast speed, strong anti-interference ability and so on. Camera Link is widely used in the field of robot vision applications. FPGA integrates rich gate circuit resources and has the advantages of high speed, which makes it suitable for processing parallel data. In this paper, a Camera Link image data processing method based on FPGA is proposed, which is verified by simulation and has been successfully applied to practical projects.

Visual Object Recognition and Pose Estimation Based on a Deep Semantic Segmentation Network

In recent years, deep learning-based object recognition algorithms become emerging in robotic vision applications. This paper addresses the design of a novel deep learning-based visual object recognition and pose estimation system for a robot manipulator to handle random object picking tasks. The proposed visual control system consists of a visual perception module, an object pose estimation module, a data argumentation module, and a robot manipulator controller. The visual perception module combines deep convolution neural networks (CNNs) and a fully connected conditional random field layer to realize an image semantic segmentation function, which can provide stable and accurate object classification results in cluttered environments. The object pose estimation module implements a model-based pose estimation method to estimate the 3D pose of the target for picking control. In addition, the proposed data argumentation module automatically generates training data for training the deep CNN. Experimental results show that the proposed scene segmentation method used in the data argumentation module reaches a high accuracy rate of 97.10% on average, which is higher than other state-of-the-art segment methods. Moreover, with the proposed data argumentation module, the visual perception module reaches an accuracy rate over than 80% and 72% in the case of detecting and recognizing one object and three objects, respectively. In addition, the proposed model-based pose estimation method provides accurate 3D pose estimation results. The average translation and rotation errors in the three axes are all smaller than 0.52 cm and 3.95 degrees, respectively. These advantages make the proposed visual control system suitable for applications of random object picking and manipulation.

Static map reconstruction and dynamic object tracking for a camera and laser scanner system

The vision‐based mobile robot's simultaneous localisation and mapping and navigation capability in dynamic environments are highly problematic elements of robot vision applications. The goal of this study is to reconstruct a static map and track the dynamic object for a camera and laser scanner system. An improved automatic calibration is designed to merge image and laser point clouds. Then, the fusion data is exploited to detect the slowly moved object and reconstruct static map. Tracking‐by‐detection requires the correct assignment of noisy detection results to object trajectories. In the proposed method, occluded regions are combined 3D motion models with object appearance to manage difficulties in crowded scenes. The proposed method was validated by experimental results gathered in a real environment and on publicly available data.