Robot Vision Research Articles

PRG4CNN: A Probabilistic Model Checking-Driven Robustness Guarantee Framework for CNNs

As an important kind of DNN (deep neural network), CNN (convolutional neural network) has made remarkable progress and been widely used in the vision and decision-making of autonomous robots. Nonetheless, in many scenarios, even a minor perturbation in input for CNNs may lead to serious errors, which means CNNs lack robustness. Formal verification is an effective method to guarantee the robustness of CNNs. Existing works predominantly concentrate on local robustness verification, which requires considerable time and space. Probabilistic robustness quantifies the robustness of CNNs, which is a practical mode of potential measurement. The state-of-the-art of probabilistic robustness verification is a test-driven approach, which is used to manually decide whether a DNN satisfies the probabilistic robustness and does not involve robustness repair. Robustness repair can improve the robustness of CNNs further. To address this issue, we propose a probabilistic model checking-driven robustness guarantee framework for CNNs, i.e., PRG4CNN. This is the first automated and complete framework for guaranteeing the probabilistic robustness of CNNs. It comprises four steps, as follows: (1) modeling a CNN as an MDP (Markov decision processes) by model learning, (2) specifying the probabilistic robustness of the CNN via the PCTL (Probabilistic Computational Tree Logic) formula, (3) verifying the probabilistic robustness with a probabilistic model checker, and (4) probabilistic robustness repair by counterexample-guided sensitivity analysis, if probabilistic robustness does not hold on the CNN. We here conduct experiments on various scales of CNNs trained on the handwriting dataset MNIST, and demonstrate the effectiveness of PRG4CNN.

Precision Monitoring of Dead Chickens and Floor Eggs with a Robotic Machine Vision Method

Modern poultry and egg production is facing challenges such as dead chickens and floor eggs in cage-free housing. Precision poultry management strategies are needed to address those challenges. In this study, convolutional neural network (CNN) models and an intelligent bionic quadruped robot were used to detect floor eggs and dead chickens in cage-free housing environments. A dataset comprising 1200 images was used to develop detection models, which were split into training, testing, and validation sets in a 3:1:1 ratio. Five different CNN models were developed based on YOLOv8 and the robot’s 360° panoramic depth perception camera. The final results indicated that YOLOv8m exhibited the highest performance, achieving a precision of 90.59%. The application of the optimal model facilitated the detection of floor eggs in dimly lit areas such as below the feeder area and in corner spaces, as well as the detection of dead chickens within the flock. This research underscores the utility of bionic robotics and convolutional neural networks for poultry management and precision livestock farming.

E2-VINS: An Event-Enhanced Visual–Inertial SLAM Scheme for Dynamic Environments

Simultaneous Localization and Mapping (SLAM) technology has garnered significant interest in the robotic vision community over the past few decades. The rapid development of SLAM technology has resulted in its widespread application across various fields, including autonomous driving, robot navigation, and virtual reality. Although SLAM, especially Visual–Inertial SLAM (VI-SLAM), has made substantial progress, most classic algorithms in this field are designed based on the assumption that the observed scene is static. In complex real-world environments, the presence of dynamic objects such as pedestrians and vehicles can seriously affect the robustness and accuracy of such systems. Event cameras, which use recently introduced motion-sensitive biomimetic sensors, efficiently capture scene changes (referred to as “events”) with high temporal resolution, offering new opportunities to enhance VI-SLAM performance in dynamic environments. Integrating this kind of innovative sensor, we propose the first event-enhanced Visual–Inertial SLAM framework specifically designed for dynamic environments, termed E2-VINS. Specifically, the system uses visual–inertial alignment strategy to estimate IMU biases and correct IMU measurements. The calibrated IMU measurements are used to assist in motion compensation, achieving spatiotemporal alignment of events. The event-based dynamicity metrics, which measure the dynamicity of each pixel, are then generated on these aligned events. Based on these metrics, the visual residual terms of different pixels are adaptively assigned weights, namely, dynamicity weights. Subsequently, E2-VINS jointly and alternately optimizes the system state (camera poses and map points) and dynamicity weights, effectively filtering out dynamic features through a soft-threshold mechanism. Our scheme enhances the robustness of classic VI-SLAM against dynamic features, which significantly enhances VI-SLAM performance in dynamic environments, resulting in an average improvement of 1.884% in the mean position error compared to state-of-the-art methods. The superior performance of E2-VINS is validated through both qualitative and quantitative experimental results. To ensure that our results are fully reproducible, all the relevant data and codes have been released.

Badminton striking motion recognition based on dense trajectory algorithm

With people paying more attention to health and quality of life, more and more people are keen to exercise through various sports, such as badminton, which is suitable for all ages. With the development of the new era, artificial intelligence and robots have gradually entered people’s lives, among which badminton robot is an intelligent robot that can compete with people in real time. The robot can not only capture and track badminton moving at high speed, but also has a high-speed motion control system to accurately complete the striking action. Therefore, the main research content of this paper is based on badminton robot vision system to capture, identify and analyze badminton players’ striking action. The main research work is as follows: A method of obtaining video segments of badminton striking action according to the flying direction and position of badminton is proposed, and a data set containing 8 kinds of common badminton striking action is made. Then, the dense trajectory algorithm is improved to recognize badminton striking action more effectively. Through the experimental study found that the use of dense trajectory algorithm for badminton players hit action recognition by only extracting the trajectory of feature points in a small range of players reduce the complexity of the algorithm but also enhance the robustness of the algorithm. Through the way of experiments to verify the effectiveness of the non-fixed length trajectory, but also improve the recognition rate of badminton striking action.

Robotics and artificial intelligence

The general objective of the research was to determine the advances related to the robotics and artificial intelligence. The specific objectives of the research are to determine the countries and organizations that successfully manufacture robots, identify the characteristics of the most representative robots and the software used in their design and operation. Methodology, in this research, 61 documents have been selected, carried out in the period 2018 - 2024; including: scientific articles, review articles and information from websites of recognized organizations. Results, the software used in the design and operation of the robots are: Robot operating systems (ros), industrial robot programming software, simulation software, robot vision software, autonomous navigation software, collaborative robot (cobot) software. Conclusions, artificial intelligence has been evolving and has managed to position itself as an important technology to improve various systems. The leading robotics producers are in Germany, Japan, and Switzerland. The organizations that successfully manufacture robots are ABB Group (Switzerland), Adept Technology Inc (United States) and Apex Automation and Robotics (Australia). And the software used in the design and operation of the robots are: Robot operating systems (ros), industrial robot programming software, simulation software, robot vision software, autonomous navigation software, collaborative robot (cobot) software.

An Enhanced Cycle Generative Adversarial Network Approach for Nighttime Pineapple Detection of Automated Harvesting Robots

Nighttime pineapple detection for automated harvesting robots is a significant challenge in intelligent agriculture. As a crucial component of robotic vision systems, accurate fruit detection is essential for round-the-clock operations. The study compared advanced end-to-end style transfer models, including U-GAT-IT, SCTNet, and CycleGAN, finding that CycleGAN produced relatively good-quality images but had issues such as the inadequate restoration of nighttime details, color distortion, and artifacts. Therefore, this study further proposed an enhanced CycleGAN approach to address limited nighttime datasets and poor visibility, combining style transfer with small-sample object detection. The improved model features a novel generator structure with ResNeXtBlocks, an optimized upsampling module, and a hyperparameter optimization strategy. This approach achieves a 29.7% reduction in FID score compared to the original CycleGAN. When applied to YOLOv7-based detection, this method significantly outperforms existing approaches, improving precision, recall, average precision, and F1 score by 13.34%, 45.11%, 56.52%, and 30.52%, respectively. These results demonstrate the effectiveness of our enhanced CycleGAN in expanding limited nighttime datasets and supporting efficient automated harvesting in low-light conditions, contributing to the development of more versatile agricultural robots capable of continuous operation.

Development of AI Model for Robotic Vision Inspection of Sheet-Metal Components in Manufacturing

This paper presents an AI-based robotic vision inspection system designed to enhance quality control in the manufacturing of sheet-metal components. The system integrates a sequential AI model with advanced image processing techniques to automate real-time defect detection and classification. Utilizing a high-resolution camera, the system captures images of components on a conveyor, which are pre- processed using grayscale conversion, Gaussian blurring, and Canny edge detection to emphasize structural details. A deep learning model then classifies isolated regions of interest based on normalized, resized images. Feature matching through ORB (Oriented FAST and Rotated BRIEF) enables accurate alignment with reference templates, while automated measurements convert pixel dimensions to physical units, ensuring reliable detection of deviations. With an average accuracy of 88.3%, the system consistently identifies subtle and complex defects, such as scratches and dimensional deviations, under variable lighting and noise conditions. This AI-driven approach reduces the need for manual inspection, minimizes error, and enhances workflow efficiency, representing a major step toward robust, real-time quality assurance solutions in industrial environments.

Fusing Events and Frames with Coordinate Attention Gated Recurrent Unit for Monocular Depth Estimation.

Monocular depth estimation is a central problem in computer vision and robot vision, aiming at obtaining the depth information of a scene from a single image. In some extreme environments such as dynamics or drastic lighting changes, monocular depth estimation methods based on conventional cameras often perform poorly. Event cameras are able to capture brightness changes asynchronously but are not able to acquire color and absolute brightness information. Thus, it is an ideal choice to make full use of the complementary advantages of event cameras and conventional cameras. However, how to effectively fuse event data and frames to improve the accuracy and robustness of monocular depth estimation remains an urgent problem. To overcome these challenges, a novel Coordinate Attention Gated Recurrent Unit (CAGRU) is proposed in this paper. Unlike the conventional ConvGRUs, our CAGRU abandons the conventional practice of using convolutional layers for all the gates and innovatively designs the coordinate attention as an attention gate and combines it with the convolutional gate. Coordinate attention explicitly models inter-channel dependencies and coordinate information in space. The coordinate attention gate in conjunction with the convolutional gate enable the network to model feature information spatially, temporally, and internally across channels. Based on this, the CAGRU can enhance the information density of the sparse events in the spatial domain in the recursive process of temporal information, thereby achieving more effective feature screening and fusion. It can effectively integrate feature information from event cameras and standard cameras, further improving the accuracy and robustness of monocular depth estimation. The experimental results show that the method proposed in this paper achieves significant performance improvements on different public datasets.

Uncertainty-Boosted Robust Video Activity Anticipation.

Video activity anticipation aims to predict what will happen in the future, embracing a broad application prospect ranging from robot vision and autonomous driving. Despite the recent progress, the data uncertainty issue, reflected as the content evolution process and dynamic correlation in event labels, has been somehow ignored. This reduces the model generalization ability and deep understanding on video content, leading to serious error accumulation and degraded performance. In this paper, we address the uncertainty learning problem and propose an uncertainty-boosted robust video activity anticipation framework, which generates uncertainty values to indicate the credibility of the anticipation results. The uncertainty value is used to derive a temperature parameter in the softmax function to modulate the predicted target activity distribution. To guarantee the distribution adjustment, we construct a reasonable target activity label representation by incorporating the activity evolution from the temporal class correlation and the semantic relationship. Moreover, we quantify the uncertainty into relative values by comparing the uncertainty among sample pairs and their temporal-lengths. This relative strategy provides a more accessible way in uncertainty modeling than quantifying the absolute uncertainty values on the whole dataset. Experiments on multiple backbones and benchmarks show our framework achieves promising performance and better robustness/interpretability.

A two-stage 6D pose estimation method for industrial textureless parts based on multi-feature fusion

Abstract 6D pose estimation is an essential supporting technology for many industrial applications such as robotic vision, human-robot collaboration and augmented reality. However, in industrial environments, due to the textureless, reflective, and occluded characteristics of industrial parts, the accuracy and adaptability to the application environment of pose estimation are limited. To solve this issue, a two-stage 6D pose estimation method for industrial parts is proposed, which uses a multi-feature fusion strategy. In the first stage, the semantic keypoints are selected to train a PVN3D-based RGBD fusion pose estimation network to predict the initial pose. In the second stage, we propose a pose iterative optimization method based on the fusion of appearance and geometric features. Experiments on the MP6D industrial dataset demonstrate that the proposed method exhibits the comparative methods. Our approach offers a novel idea for accurate and robust pose estimation of industrial parts.

Reinforced Multi-Path Graph Attention Network (MPGA-RL-Net) for Adaptive Robotic Vision with Task-Specific Region Prioritization

Reinforced Multi-Path Graph Attention Network (MPGA-RL-Net) for Adaptive Robotic Vision with Task-Specific Region Prioritization

Online Active Continual Learning for Robotic Lifelong Object Recognition.

In real-world applications, robotic systems collect vast amounts of new data from ever-changing environments over time. They need to continually interact and learn new knowledge from the external world to adapt to the environment. Particularly, lifelong object recognition in an online and interactive manner is a crucial and fundamental capability for robotic systems. To meet this realistic demand, in this article, we propose an online active continual learning (OACL) framework for robotic lifelong object recognition, in the scenario of both classes and domains changing with dynamic environments. First, to reduce the labeling cost as much as possible while maximizing the performance, a new online active learning (OAL) strategy is designed by taking both the uncertainty and diversity of samples into account to protect the information volume and distribution of data. In addition, to prevent catastrophic forgetting and reduce memory costs, a novel online continual learning (OCL) algorithm is proposed based on the deep feature semantic augmentation and a new loss-based deep model and replay buffer update, which can mitigate the class imbalance between the old and new classes and alleviate confusion between two similar classes. Moreover, the mistake bound of the proposed method is analyzed in theory. OACL allows robots to select the most representative new samples to query labels and continually learn new objects and new variants of previously learned objects from a nonindependent and identically distributed (i.i.d.) data stream without catastrophic forgetting. Extensive experiments conducted on real lifelong robotic vision datasets demonstrate that our algorithm, even trained with fewer labeled samples and replay exemplars, can achieve state-of-the-art performance on OCL tasks.

An effective inliers selection strategy for high-precision image feature matching

ABSTRACT Seeking a reliable inliers selection method to image feature matching is a fundamental and important task in image processing and robotic vision. To improve the precision of feature matching on putative matches with heavy outliers, we propose a new strategy to select the inliers from the rough feature correspondence-set. Firstly, we construct the inliers selection problem as a concise mathematical model with linear time complexity based on the local K-nearest neighbour structure preserving (KNNSP). Then, to improve the robustness of this model, we design a threshold decrease strategy for this model to iteratively solve the optimal inliers set. Extensive experiments on various image pairs demonstrate that the proposed method does not need prior information and can be directly used for rigid or non-rigid image matching problems. In addition, compared with other typical image matching algorithms, our method ensures competitive performance in precision and recall.

A Dynamic Interference Detection Method of Underwater Scenes Based on Deep Learning and Attention Mechanism.

Improving the three-dimensional reconstruction of underwater scenes is a challenging and hot topic in the field of underwater robot vision system research. High dynamic interference underwater has always been one of the key issues affecting the 3D reconstruction of underwater scenes. However, due to the complex underwater environment and insufficient light, existing target detection algorithms cannot meet the requirements. This paper uses the YOLOv8 network as the basis of the algorithm and proposes an underwater dynamic target detection algorithm based on improved YOLOv8. This algorithm first improves the feature extraction layer of the YOLOv8 network, improves the convolutional network structure of Bottleneck, reduces the amount of calculation and improves detection accuracy. Secondly, it adds an improved SE attention mechanism to make the network have a better feature extraction effect; in addition, the confidence box loss function of the network is improved, and the CIoU loss function is replaced by the MPDIoU loss function, which effectively improves the model convergence speed. Experimental results show that the mAP value of the improved YOLOv8 underwater dynamic target detection algorithm proposed in this article can reach 95.1%, and it can detect underwater dynamic targets more accurately, especially small dynamic targets in complex underwater scenes.

The application of Deep Learning in the field of Robotics

Abstract. As a new field with rapid development in the past two decades, deep learning has received more and more attention from researchers. Neural network adopts widely interconnected structure and effective learning mechanisms to simulate the process of information processing in the human brain, which is an important method in the development of artificial intelligence. Compared with shallow models, deep learning can achieve stronger unsupervised autonomous learning capabilities by increasing the number of layers of the network. In the past, the robots needed to be controlled by the humans; they controlled the robot to finish different tasks. In recent years, the researchers have tried to apply deep learning in the field of robots. Deep neural networks can simulate the complex cognitive laws of the human brain, so many researchers apply deep learning to robots to make robots have the thinking ability of humans. This article will review the cases of combining robot and in-depth learning in recent decades, Summarizing the achievements and problems faced in the three fields of robot vision, trajectory planning and motion control.

Enhancing target recognition rate in atmospheric turbulence using orbital angular momentum spectra of vortex beams

Abstract Traditional methods for extracting and recognizing targets from laser echo signals typically involve complex processing and require extensive data. Vortex beams carry orbital angular momentum (OAM), and upon reflection from a target, the distribution of the OAM spectrum carries features related to the target, thereby enriching the dimensions of target recognition. Using the OAM spectrum simplifies the recognition process but faces challenges like atmospheric turbulence that affect beam transmission and target recognition accuracy. Our study employs the Gerchberg–Saxton phase retrieval (GS) algorithm to mitigate the effects of atmospheric turbulence on the beams. Using OAM spectrum data, we achieved effective target recognition with various shapes under atmospheric turbulence through a back-propagation neural network (BPNN). Simulations revealed a recognition rate increase from 76.25% to 96% post-compensation by the GS algorithm. We also found that the highest recognition rate occurs at a target ratio of 0.2. After compensation with the GS algorithm at a target ratio of 0.1, the recognition rate for each shape increased to 99%. This demonstrates the effectiveness of utilizing the OAM spectrum for recognizing diverse target shapes, with the GS algorithm further improving recognition rates. These findings can be applied to intelligent transportation and robotic vision.

Inspiration from Visual Ecology for Advancing Multifunctional Robotic Vision Systems: Bio-inspired Electronic Eyes and Neuromorphic Image Sensors.

In robotics, particularly for autonomous navigation and human-robot collaboration, the significance of unconventional imaging techniques and efficient data processing capabilities is paramount. The unstructured environments encountered by robots, coupled with complex missions assigned to them, present numerous challenges necessitating diverse visual functionalities, and consequently, the development of multifunctional robotic vision systems has become indispensable. Meanwhile, rich diversity inherent in animal vision systems, honed over evolutionary epochs to meet their survival demands across varied habitats, serves as a profound source of inspirations. Here, recent advancements in multifunctional robotic vision systems drawing inspiration from natural ocular structures and their visual perception mechanisms are delineated. First, unique imaging functionalities of natural eyes across terrestrial, aerial, and aquatic habitats and visual signal processing mechanism of humans are explored. Then, designs and functionalities of bio-inspired electronic eyes are explored, engineered to mimic key components and underlying optical principles of natural eyes. Furthermore, neuromorphic image sensors are discussed, emulating functional properties of synapses, neurons, and retinas and thereby enhancing accuracy and efficiency of robotic vision tasks. Next, integration examples of electronic eyes with mobile robotic/biological systems are introduced. Finally, a forward-looking outlook on the development of bio-inspired electronic eyes and neuromorphic image sensors is provided.

Revolutionizing endometriosis treatment: automated surgical operation through artificial intelligence and robotic vision.

Clinical limitations due to poverty significantly impact the lives and health of many individuals globally. Nevertheless, this challenge can be addressed with modern technologies, particularly through robotics and artificial intelligence. This study aims to address these challenges using advanced technologies in robotic surgery and artificial intelligence, proposing a method to fully automate endometriosis robotic surgery with a focus on interpretability, accuracy, and reliability. A methodology for fully automatic endometriosis surgery is introduced. Given the complexity of endometriosis lesions detection, they are categorized by their anatomical location to improve system interpretability. Then, three ensemble U-Net frameworks are designed to detect and localize common types of endometriosis lesions intraoperatively. A cross-training approach is employed, exploring U-Net models with diverse neural architectures-such as ResNet50, ResNet101, VGG19, InceptionV3, MobileNet, and EfficientNetB7-to develop U-Net ensemble models for precise endometriosis lesions segmentation. A novel image augmentation technique is also introduced, enhancing the segmentation models' accuracy and reliability. Furthermore, two U-Net models are developed to localize the ovaries and uterus, mitigating unexpected noise and bolstering the method's accuracy and reliability. The image segmentation models, assessed using the Intersection over Union (IoU) metric, achieved outstanding results: 97.57% for ovarian, 96.35% for uterine, and 92.58% for peritoneal endometriosis. This study proposes a fully automatic method for some common types of endometriosis surgery, including ovarian endometriomas and superficial endometriosis. This method is centered around three ensemble U-Net frameworks and a noise reduction technique using two additional U-Nets for localizing the ovaries and uterus. This approach has the potential to significantly improve the accuracy and reliability of robotic surgeries, potentially reducing healthcare costs and improving outcomes for patients worldwide.

Cascade contour-enhanced panoptic segmentation for robotic vision perception.

Panoptic segmentation plays a crucial role in enabling robots to comprehend their surroundings, providing fine-grained scene understanding information for robots' intelligent tasks. Although existing methods have made some progress, they are prone to fail in areas with weak textures, small objects, etc. Inspired by biological vision research, we propose a cascaded contour-enhanced panoptic segmentation network called CCPSNet, attempting to enhance the discriminability of instances through structural knowledge. To acquire the scene structure, a cascade contour detection stream is designed, which extracts comprehensive scene contours using channel regulation structural perception module and coarse-to-fine cascade strategy. Furthermore, the contour-guided multi-scale feature enhancement stream is developed to boost the discrimination ability for small objects and weak textures. The stream integrates contour information and multi-scale context features through structural-aware feature modulation module and inverse aggregation technique. Experimental results show that our method improves accuracy on the Cityscapes (61.2 PQ) and COCO (43.5 PQ) datasets while also demonstrating robustness in challenging simulated real-world complex scenarios faced by robots, such as dirty cameras and rainy conditions. The proposed network promises to help the robot perceive the real scene. In future work, an unsupervised training strategy for the network could be explored to reduce the training cost.

Three-dimensional Reconstruction of Grassland Landforms Based on Intelligent Robot Vision System and Numerical Simulation of Its Characteristics

Three-dimensional Reconstruction of Grassland Landforms Based on Intelligent Robot Vision System and Numerical Simulation of Its Characteristics