Robot Visual Perception Research Articles

MRFTrans: Multimodal Representation Fusion Transformer for monocular 3D semantic scene completion

The complete understanding of 3D scenes is crucial in robotic visual perception, impacting tasks such as motion planning and map localization. However, due to the limited field of view and scene occlusion constraints of sensors, inferring complete scene geometry and semantic information from restricted observations is challenging. In this work, we propose a novel Multimodal Representation Fusion Transformer framework (MRFTrans) that robustly fuses semantic, geometric occupancy, and depth representations for monocular-image-based scene completion. MRFTrans centers on an affinity representation fusion transformer, integrating geometric occupancy and semantic relationships within a transformer architecture. This integration enables the modeling of long-range dependencies within scenes for inferring missing information. Additionally, we present a depth representation fusion method, efficiently extracting reliable depth knowledge from biased monocular estimates. Extensive experiments demonstrate MRFTrans’s superiority, setting a new benchmark on SemanticKITTI and NYUv2 datasets. It significantly enhances completeness and accuracy, particularly in large structures, movable objects, and scene components with major occlusions. The results underscore the benefits of the affinity-aware transformer and robust depth fusion in monocular-image-based completion.

A novel jujube tree trunk and branch salient object detection method for catch-and-shake robotic visual perception

Visual perception has become a prerequisite for automated jujube harvesting robot operations under complex orchard conditions. Catch-and-Shake harvesting, as the most efficient and common harvesting method, has widely been applied on various manually operated harvesters to complete large-area jujube fruit harvesting. However, the main factors restricting the development of existing harvesters are labor shortage, high labor cost, and low operating efficiency. To address the issues, we designed a catch-and-shake harvesting robot for jujube tree trunks and branches visual perception that can provide a barrier-free catch-and-shake operation area and guide the manipulator to reach the area to complete the harvesting operation. Meanwhile, a visual perception system including tree trunks and branches detection, skeleton extraction, catch-and-shake area confirmation was presented to guide robot intelligent operations. In the visual perception system, a novel salientobjectdetectionmodel called feature intersection and fusion Transformer (FIT-Transformer) network was proposed to split branches and background to provide reference for determining safe catch-and-shake areas. Moreover, we designed a diverse feature aggregation (DFA) and an attention feature fusion module (AFFM) to strengthen feature learning capabilities and obtain robust perception models. Comparative experimental results showed that our proposed FIT-Transformer model outperformed 12 state-of-the-art (SOTA) algorithms including C2FNet, RAS, BASNet, U2Net, SCRNet, PiCANet, EDRNet, EGNet, ICONR, VST, TransSOD and ABiU_Net. Specifically, the segmentation accuracy of jujube tree trunks and branches using our method showed the satisfactory result on five evaluation indexes under natural environment (the EM, SM, WF, FM and MAE reached 0.9713, 0.8991, 0.8854, 0.8905, and 0.0302, respectively). Field experiments also proved that our method could meet the requirements of operational accuracy and real-time operations.

Panoramic visual system for spherical mobile robots

Abstract Aimed at the challenges of wide-angle mobile robot visual perception for diverse field applications, we present the spherical robot visual system that uses a 360° field of view (FOV) for realizing real-time object detection. The spherical robot image acquisition system model is developed with optimal parameters, including camera spacing, camera axis angle, and the distance of the target image plane. Two 180 $^{\circ}$ -wide panoramic FOVs, front and rear view, are formed using four on-board cameras. The speed of the SURF algorithm is increased for feature extraction and matching. For seamless fusion of the images, an improved fade-in and fade-out algorithm is used, which not only improves the seam quality but also improves object detection performance. The speed of the dynamic image stitching is significantly enhanced by using a cache-based sequential image fusion method. On top of the acquired panoramic wide FOVs, the YOLO algorithm is used for real-time object detection. The panoramic visual system for the spherical robot is then tested in real time, which outputs panoramic views of the scene at an average frame rate of 21.69 fps and panoramic views with object detection at an average of 15.39 fps.

Visual Sensing and Depth Perception for Welding Robots and Their Industrial Applications.

With the rapid development of vision sensing, artificial intelligence, and robotics technology, one of the challenges we face is installing more advanced vision sensors on welding robots to achieve intelligent welding manufacturing and obtain high-quality welding components. Depth perception is one of the bottlenecks in the development of welding sensors. This review provides an assessment of active and passive sensing methods for depth perception and classifies and elaborates on the depth perception mechanisms based on monocular vision, binocular vision, and multi-view vision. It explores the principles and means of using deep learning for depth perception in robotic welding processes. Further, the application of welding robot visual perception in different industrial scenarios is summarized. Finally, the problems and countermeasures of welding robot visual perception technology are analyzed, and developments for the future are proposed. This review has analyzed a total of 2662 articles and cited 152 as references. The potential future research topics are suggested to include deep learning for object detection and recognition, transfer deep learning for welding robot adaptation, developing multi-modal sensor fusion, integrating models and hardware, and performing a comprehensive requirement analysis and system evaluation in collaboration with welding experts to design a multi-modal sensor fusion architecture.

Flexible optoelectronic synaptic transistors for neuromorphic visual systems

Neuromorphic visual systems that integrate the functionalities of sensing, memory, and processing are expected to overcome the shortcomings of conventional artificial visual systems, such as data redundancy, data access delay, and high-energy consumption. Neuromorphic visual systems based on emerging flexible optoelectronic synaptic devices have recently opened up innovative applications, such as robot visual perception, visual prosthetics, and artificial intelligence. Various flexible optoelectronic synaptic devices have been fabricated, which are either two-terminal memristors or three-terminal transistors. In flexible optoelectronic synaptic transistors (FOSTs), the synaptic weight can be modulated by the electricity and light synergistically, which endows the neuromorphic visual systems with versatile functionalities. In this Review, we present an overview of the working mechanisms, device structures, and active materials of FOSTs. Their applications in neuromorphic visual systems for color recognition, image recognition and memory, motion detection, and pain perception are presented. Perspectives on the development of FOSTs are finally outlined.

A Novel Visible-Depth-Thermal Image Dataset of Salient Object Detection for Robotic Visual Perception

Visual perception plays an important role in industrial information field, especially in robotic grasping application. In order to detect the object to be grasped quickly and accurately, salient object detection (SOD) is employed to the above task. Although the existing SOD methods have achieved impressive performance, they still have some limitations in the complex interference environment of practical application. To better deal with the complex interference environment, a novel triple-modal images fusion strategy is proposed to implement SOD for robotic visual perception, namely visible-depth-thermal (VDT) SOD. Meanwhile, we build an image acquisition system under variable lighting scene and construct a novel benchmark dataset for VDT SOD (VDT-2048 dataset). Multiple modal images will be introduced to assist each other to highlight the salient regions. But, inevitably, interference will also be introduced. In order to achieve effective cross-modal feature fusion while suppressing information interference, a hierarchical weighted suppress interference (HWSI) method is proposed. The comprehensive experimental results prove that our method achieves better performance than the state-of-the-art methods.

A novel robotic visual perception framework for underwater operation

A novel robotic visual perception framework for underwater operation

Complex Sports Target Tracking with Machine Learning: Take Basketball as an Example

Object tracking is a hot issue in vision technology research; it is used in scenarios like intelligent monitoring, autonomous driving, and robot visual perception. With the rapid development of sports, tracking of targets in complex sports scenes represented by basketball and football has gradually attracted attention. The target tracking algorithm based on machine learning (ML) has been gradually proposed. With the powerful feature extraction for convolutional neural network (CNN), it greatly improves accuracy and has better robustness in the face of complex sports scenes. However, the tracking algorithm based on deep learning has many network layers and parameters, which makes training and update speed of model slower. In this regard, taking basketball as an example, this paper designs a low-parameter deep learning-based complex sports target tracking algorithm, which greatly reduces the size of the model while ensuring the tracking accuracy. Aiming at the problem of large number of parameters and large model of deep learning tracking algorithm, this work proposes a network structure model with asymmetric convolution module. The asymmetric convolution module includes two convolutional layers, the compression layer and the asymmetric layer. To improve accuracy, this work designs a new triplet loss. Compared with original logistic loss, triplet loss function can fully utilize the latent relationship between the inputs, so that the network model can obtain higher tracking accuracy. Finally, this paper proposes a low-parameter deep learning-based target tracking algorithm combining asymmetric convolution and triple loss function. Comprehensive and systematic experiments demonstrate the effectiveness of this work in tracking complex sports objects.

JRDB: A Dataset and Benchmark of Egocentric Robot Visual Perception of Humans in Built Environments.

We present JRDB, a novel egocentric dataset collected from our social mobile manipulator JackRabbot. The dataset includes 64 minutes of annotated multimodal sensor data including stereo cylindrical 360 ° RGB video at 15 fps, 3D point clouds from two 16 planar rays Velodyne LiDARs, line 3D point clouds from two Sick Lidars, audio signal, RGB-D video at 30 fps, 360 ° spherical image from a fisheye camera and encoder values from the robot's wheels. Our dataset incorporates data from traditionally underrepresented scenes such as indoor environments and pedestrian areas, all from the ego-perspective of the robot, both stationary and navigating. The dataset has been annotated with over 2.4 million bounding boxes spread over five individual cameras and 1.8 million associated 3D cuboids around all people in the scenes totaling over 3500 time consistent trajectories. Together with our dataset and the annotations, we launch a benchmark and metrics for 2D and 3D person detection and tracking. With this dataset, which we plan on extending with further types of annotation in the future, we hope to provide a new source of data and a test-bench for research in the areas of egocentric robot vision, autonomous navigation, and all perceptual tasks around social robotics in human environments.

Complex environment perception and positioning based visual information retrieval

The biological vision model is devoted to provide a novel technology approach by merging new cognitive visual features with inspired nerve cells cognitive intelligence cortex and try to relate with real worlds object recognition. To perceive an arbitrary natural scene from complex environment perception and sensing in robotic mobility and manipulation on unstructured random natural scene understanding is a challenging problem in the visual image processing. This paper has considered neural network (NN) which is nothing but the grid of “neurons like” nodes. Based on the NN technique,the authors have proposed a new scheme for the scene understanding and recognition. In addition, the significant intellectual visual features are also incorporated for scene expression; those are very crucial and provide cognitive intelligence to robot vision. Due to the dynamic nature of artificial neural network intelligence, we have adapted the attributes of the Gabor filter and Laplacian of Gaussian filter; those play the significant role in the robot visual perception. Through the study of perceptual ability of the natural scene image from complex environment for robot vision is enhanced with the integration of cognitive visual features and the scene expression.

Three-dimensional reconstruction method based on bionic active sensing in precision assembly.

With the prevailing application of new materials and the higher requirements for the quality and efficiency of production in the equipment manufacturing industry, traditional assembly methods can hardly meet the needs of large-scale production, especially in the field of high-precision assembly. Robot assembly guided by visual perception has become the key of the research in the field of engineering technology. It requires higher accuracy of robot visual perception and the control over force, position and so on. However, in 3C assembly, most products are made of transparent materials such as glass. Because of the transparency and specular reflection of the surface, 3D reconstruction of transparent objects is a very difficult problem in computer vision, in that the traditional visual perception methods could not be accurate enough. The present research proposes a bionic active sensing algorithm for 3D perception and reconstruction and realizes high-precision 3D by applying the registration algorithm. The purpose is to solve the problems existing in the traditional visual perception method, such as difficulties in achieving active sensing, low accuracy of point clouds registration, and complex computation. The results of the experiments show that the present method is efficient and accurate in 3D reconstruction. It reduces the planar reconstruction error to 0.064 mm and the surface reconstruction error to 0.177 mm.

Rigid 3D Registration: A Simple Method Free of SVD and Eigen-Decomposition

The 3-D registration is extensively required in many industrial applications involving 3-D perception and reconstruction. However, conventional methods consisting of singular value decomposition (SVD) or eigendecomposition are all hard to be implemented and are difficult to be ported using simple digital circuit prototypes. A novel solution is obtained to solve the rigid 3-D registration problem, motivated by previous eigendecomposition approaches. Different from existing solvers, the proposed algorithm does not require sophisticated matrix operations, e.g., SVD or eigenvalue decomposition. Instead, the optimal eigenvector of the point cross-covariance matrix can be computed within several iterations. It is also proven that the optimal rotation matrix can be directly computed for cases without the need for a quaternion. The simple framework provides a very easy approach, even without floating-number processing. Simulations on noise-corrupted point clouds have verified the robustness and computation speed of the proposed method. The final results indicate that the proposed algorithm is accurate, robust, and owns over 60%–80% less computation time than representatives. It has also been applied to real-world applications for faster robotic visual perception.

Visual recognition for humanoid robots

Visual perception is a fundamental component for most robotics systems operating in human environments. Specifically, visual recognition is a prerequisite to a large variety of tasks such as tracking, manipulation, human–robot interaction. As a consequence, the lack of successful recognition often becomes a bottleneck for the application of robotics system to real-world situations. In this paper we aim at improving the robot visual perception capabilities in a natural, human-like fashion, with a very limited amount of constraints to the acquisition scenario. In particular our goal is to build and analyze a learning system that can rapidly be re-trained in order to incorporate new evidence if available. To this purpose, we review the state-of-the-art coding–pooling pipelines for visual recognition and propose two modifications which allow us to improve the quality of the representation, while maintaining real-time performances: a coding scheme, Best Code Entries (BCE), and a new pooling operator, Mid-Level Classification Weights (MLCW). The former focuses entirely on sparsity and improves the stability and computational efficiency of the coding phase, the latter increases the discriminability of the visual representation, and therefore the overall recognition accuracy of the system, by exploiting data supervision. The proposed pipeline is assessed from a qualitative perspective on a Human–Robot Interaction (HRI) application on the iCub platform. Quantitative evaluation of the proposed system is performed both on in-house robotics data-sets (iCubWorld) and on established computer vision benchmarks (Caltech-256, PASCAL VOC 2007). As a byproduct of this work, we provide for the robotics community an implementation of the proposed visual recognition pipeline which can be used as perceptual layer for more complex robotics applications.

Perceptual learning and abstraction in machine learning: an application to autonomous robotics

This paper deals with the possible benefits of perceptual learning in artificial intelligence. On the one hand, perceptual learning is more and more studied in neurobiology and is now considered as an essential part of any living system. In fact, perceptual learning and cognitive learning are both necessary for learning and often depend on each other. On the other hand, many works in machine learning are concerned with in order to reduce the amount of complexity related to some learning tasks. In the abstraction framework, perceptual learning can be seen as a specific process that learns how to transform the data before the traditional learning task itself takes place. In this paper, we argue that biologically inspired perceptual learning mechanisms could be used to build efficient low-level abstraction operators that deal with real-world data. To illustrate this, we present an application where perceptual-learning-inspired metaoperators are used to perform an abstraction on an autonomous robot visual perception. The goal of this work is to enable the robot to learn how to identify objects it encounters in its environment.

A model of abstraction in visual perception

In artificial intelligence, abstraction has been mainly studied as a mapping between languages in relation to problem-solving, with the aim of reducing the complexity of the task. However, abstraction has a much larger scope in reasoning; we are investigating, in this article, how abstraction can be used in concept representation. To this aim, we propose a novel, perception-based model of abstraction, which originates from the observation that conceptualization of a domain, even though involving entities belonging to several epistemological levels, is nevertheless primarily based on perception. This view has been recently advocated by Goldstone and Barsalou in cognitive science. A model of representation/abstraction is then proposed and its application to a real-world problem of robot visual perception and categorization is presented.

Visual Perception of Robot Environment – Construction Methods and Means

Methods for investigating and constructing the robot visual systems are discussed. These methods are developed for creation of adaptive robots, automatization of assembly, inspection and preorganizing of environment the robots are Operating in.Specific features of robot visual perception and their employment for 3-D objects recognition are analysed.Results of research in hardware and software of CAD system for the adaptive robots are presented.