Robot Vision System Research Articles

A Computationally Efficient Neuronal Model for Collision Detection with Contrast Polarity-Specific Feed-Forward Inhibition

Animals utilize their well-evolved dynamic vision systems to perceive and evade collision threats. Driven by biological research, bio-inspired models based on lobula giant movement detectors (LGMDs) address certain gaps in constructing artificial collision-detecting vision systems with robust selectivity, offering reliable, low-cost, and miniaturized collision sensors across various scenes. Recent progress in neuroscience has revealed the energetic advantages of dendritic arrangements presynaptic to the LGMDs, which receive contrast polarity-specific signals on separate dendritic fields. Specifically, feed-forward inhibitory inputs arise from parallel ON/OFF pathways interacting with excitation. However, none of the previous research has investigated the evolution of a computational LGMD model with feed-forward inhibition (FFI) separated by opposite polarity. This study fills this vacancy by presenting an optimized neuronal model where FFI is divided into ON/OFF channels, each with distinct synaptic connections. To align with the energy efficiency of biological systems, we introduce an activation function associated with neural computation of FFI and interactions between local excitation and lateral inhibition within ON/OFF channels, ignoring non-active signal processing. This approach significantly improves the time efficiency of the LGMD model, focusing only on substantial luminance changes in image streams. The proposed neuronal model not only accelerates visual processing in relatively stationary scenes but also maintains robust selectivity to ON/OFF-contrast looming stimuli. Additionally, it can suppress translational motion to a moderate extent. Comparative testing with state-of-the-art based on ON/OFF channels was conducted systematically using a range of visual stimuli, including indoor structured and complex outdoor scenes. The results demonstrated significant time savings in silico while retaining original collision selectivity. Furthermore, the optimized model was implemented in the embedded vision system of a micro-mobile robot, achieving the highest success ratio of collision avoidance at 97.51% while nearly halving the processing time compared with previous models. This highlights a robust and parsimonious collision-sensing mode that effectively addresses real-world challenges.

Three-dimensional Reconstruction of Grassland Landforms Based on Intelligent Robot Vision System and Numerical Simulation of Its Characteristics

Three-dimensional Reconstruction of Grassland Landforms Based on Intelligent Robot Vision System and Numerical Simulation of Its Characteristics

Inspiration from Visual Ecology for Advancing Multifunctional Robotic Vision Systems: Bio-inspired Electronic Eyes and Neuromorphic Image Sensors.

In robotics, particularly for autonomous navigation and human-robot collaboration, the significance of unconventional imaging techniques and efficient data processing capabilities is paramount. The unstructured environments encountered by robots, coupled with complex missions assigned to them, present numerous challenges necessitating diverse visual functionalities, and consequently, the development of multifunctional robotic vision systems has become indispensable. Meanwhile, rich diversity inherent in animal vision systems, honed over evolutionary epochs to meet their survival demands across varied habitats, serves as a profound source of inspirations. Here, recent advancements in multifunctional robotic vision systems drawing inspiration from natural ocular structures and their visual perception mechanisms are delineated. First, unique imaging functionalities of natural eyes across terrestrial, aerial, and aquatic habitats and visual signal processing mechanism of humans are explored. Then, designs and functionalities of bio-inspired electronic eyes are explored, engineered to mimic key components and underlying optical principles of natural eyes. Furthermore, neuromorphic image sensors are discussed, emulating functional properties of synapses, neurons, and retinas and thereby enhancing accuracy and efficiency of robotic vision tasks. Next, integration examples of electronic eyes with mobile robotic/biological systems are introduced. Finally, a forward-looking outlook on the development of bio-inspired electronic eyes and neuromorphic image sensors is provided.

Excellent tomato detector based on pruning and distillation to balance accuracy and lightweight

Rapid and accurate detection of tomatoes in natural plantation environments is the linchpin for smart harvesting. However, the complexity of existing models necessitates high hardware computing power, constraining the deployment of tomato recognition models in picking robots. In this study, the overall scheme framework of the vision algorithm is divided into three stages. In the first stage, based on YOLOv8s, a small object detection layer was added along with hyperparameter optimization, resulting in the proposed YOLOv8-TomatoTeacher (YOLOv8-TT) to improve the accuracy and precise positioning of tomato detection, laying the foundation for subsequent operations. In the second stage, the pruning algorithm is applied to the trained YOLOv8-TT model to trim and optimize redundant channels, yielding the pruned lightweight YOLOv8-TomatoStudent (YOLOv8-TS) model. In the third stage, using YOLOv8-TT as the teacher model and YOLOv8-TS as the student model, the detection accuracy of YOLOv8-TS was further enhanced through knowledge distillation, thus obtaining the YOLOv8-TomatoDetector (YOLOv8-TD) that considers both detection speed and accuracy. Experimental results indicate that YOLOv8-TD demonstrates significant advantages in parameter quantity and model complexity, with only 0.35 M Params and 4.4G GFlops, which are respectively reduced by 96.86 % and 84.51 % compared with baseline model YOLOv8s. In addition, YOLOv8-TD achieved a 1.4 % increase in mAP, and the detection speed of 354f/s satisfies real-time requirements. The model proposed in this study can swiftly and accurately detect dense small tomato targets, reducing the deployment cost and difficulty of the robot vision system.

Implementing a Vision-Based ROS Package for Reliable Part Localization and Displacement from Conveyor Belts

The use of computer vision in the industry has become fundamental, playing an essential role in areas such as quality control and inspection, object recognition/tracking, and automation. Despite this constant growth, robotic cell systems employing computer vision encounter significant challenges, such as a lack of flexibility to adapt to different tasks or types of objects, necessitating extensive adjustments each time a change is required. This highlights the importance of developing a system that can be easily reused and reconfigured to address these challenges. This paper introduces a versatile and adaptable framework that exploits Computer Vision and the Robot Operating System (ROS) to facilitate pick-and-place operations within robotic cells, offering a comprehensive solution for handling and sorting random-flow objects on conveyor belts. Designed to be easily configured and reconfigured, it accommodates ROS-compatible robotic arms and 3D vision systems, ensuring adaptability to different technological requirements and reducing deployment costs. Experimental results demonstrate the framework’s high precision and accuracy in manipulating and sorting tested objects. Thus, this framework enhances the efficiency and flexibility of industrial robotic systems, making object manipulation more adaptable for unpredictable manufacturing environments.

Feline eye-inspired artificial vision for enhanced camouflage breaking under diverse light conditions.

Biologically inspired artificial vision research has led to innovative robotic vision systems with low optical aberration, wide field of view, and compact form factor. However, challenges persist in object detection and recognition against complex backgrounds and varied lighting. Inspired by the feline eye, which features a vertically elongated pupil and tapetum lucidum, this study introduces an artificial vision system designed for superior object detection and recognition in a monocular framework. Using a slit-like elliptical aperture and a patterned metal reflector beneath a hemispherical silicon photodiode array, the system reduces excessive light and enhances photosensitivity. This design achieves clear focus under bright light and enhanced sensitivity in dim conditions. Theoretical and experimental analyses demonstrate the system's ability to filter redundant information and detect camouflaged objects in diverse lighting, representing a substantial advancement in monocular camera technology and the potential of biomimicry in optical innovations.

Open-Ended Online Learning for Autonomous Visual Perception.

The visual perception systems aim to autonomously collect consecutive visual data and perceive the relevant information online like human beings. In comparison with the classical static visual systems focusing on fixed tasks (e.g., face recognition for visual surveillance), the real-world visual systems (e.g., the robot visual system) often need to handle unpredicted tasks and dynamically changed environments, which need to imitate human-like intelligence with open-ended online learning ability. Therefore, we provide a comprehensive analysis of open-ended online learning problems for autonomous visual perception in this survey. Based on "what to online learn" among visual perception scenarios, we classify the open-ended online learning methods into five categories: instance incremental learning to handle data attributes changing, feature evolution learning for incremental and decremental features with the feature dimension changed dynamically, class incremental learning and task incremental learning aiming at online adding new coming classes/tasks, and parallel and distributed learning for large-scale data to reveal the computational and storage advantages. We discuss the characteristic of each method and introduce several representative works as well. Finally, we introduce some representative visual perception applications to show the enhanced performance when using various open-ended online learning models, followed by a discussion of several future directions.

The Integration of Robotic Arm and Vision System With Magnetic Tractor Beam Control for Precision Catheter Manipulation in Medical Procedures

The Integration of Robotic Arm and Vision System With Magnetic Tractor Beam Control for Precision Catheter Manipulation in Medical Procedures

Enhanced Real-Time Target Detection for Picking Robots Using Lightweight CenterNet in Complex Orchard Environments

The rapid development of artificial intelligence and remote sensing technologies is indispensable for modern agriculture. In orchard environments, challenges such as varying light conditions and shading complicate the tasks of intelligent picking robots. To enhance the recognition accuracy and efficiency of apple-picking robots, this study aimed to achieve high detection accuracy in complex orchard environments while reducing model computation and time consumption. This study utilized the CenterNet neural network as the detection framework, introducing gray-centered RGB color space vertical decomposition maps and employing grouped convolutions and depth-separable convolutions to design a lightweight feature extraction network, Light-Weight Net, comprising eight bottleneck structures. Based on the recognition results, the 3D coordinates of the picking point were determined within the camera coordinate system by using the transformation relationship between the image’s physical coordinate system and the camera coordinate system, along with depth map distance information of the depth map. Experimental results obtained using a testbed with an orchard-picking robot indicated that the proposed model achieved an average precision (AP) of 96.80% on the test set, with real-time performance of 18.91 frames per second (FPS) and a model size of only 17.56 MB. In addition, the root-mean-square error of positioning accuracy in the orchard test was 4.405 mm, satisfying the high-precision positioning requirements of the picking robot vision system in complex orchard environments.

Algorithmic Implementation of Visually Guided Interceptive Actions: Harmonic Ratios and Stimulation Invariants

This research presents a novel algorithmic implementation to improve the analysis of visually controlled interception and accompanying motor action through the computational application of harmonic ratios and stimulation invariants. Unlike traditional models that focus mainly on psychological aspects, our approach integrates the relevant constructs into a practical mathematical framework. This allows for dynamic prediction of interception points with improved accuracy and real-time perception–action capabilities, essential for applications in neurorehabilitation and virtual reality. Our methodology uses stimulation invariants as key parameters within a mathematical model to quantitatively predict and improve interception outcomes. The results demonstrate the superior performance of our algorithms over conventional methods, confirming their potential for advancing robotic vision systems and adaptive virtual environments. By translating complex theories of visual perception into algorithmic solutions, this study provides innovative ways to improve motion perception and interactive systems. This study aims to articulate the complex interplay of geometry, perception, and technology in understanding and utilizing cross ratios at infinity, emphasizing their practical applications in virtual and augmented reality settings.

Advancing the Robotic Vision Revolution: Development and Evaluation of a Bionic Binocular System for Enhanced Robotic Vision.

This paper describes a novel bionic eye binocular vision system designed to mimic the natural movements of the human eye. The system provides a broader field of view and enhances visual perception in complex environments. Compared with similar bionic binocular cameras, the JEWXON BC200 bionic binocular camera developed in this study is more miniature. It consumes only 2.8 W of power, which makes it ideal for mobile robots. Combining axis and camera rotation enables more seamless panoramic image synthesis and is therefore suitable for self-rotating bionic binocular cameras. In addition, combined with the YOLO-V8 model, the camera can accurately recognize objects such as clocks and keyboards. This research provides new ideas for the development of robotic vision systems.

Effective recognition design in 8-ball billiards vision systems for training purposes based on Xception network modified by improved Chaos African Vulture Optimizer

This research paper presents a comprehensive investigation into the utilization of color image processing technologies and deep learning algorithms in the development of a robot vision system specifically designed for 8-ball billiards. The sport of billiards, with its various games and ball arrangements, presents unique challenges for robotic vision systems. The proposed methodology addresses these challenges through two main components: object detection and ball pattern recognition. Initially, a robust algorithm is employed to detect the billiard balls using color space transformation and thresholding techniques. This is followed by determining the position of the billiard table through strategic cropping and isolation of the primary table area. The crucial phase involves the intricate task of recognizing ball patterns to differentiate between solid and striped balls. To achieve this, a modified convolutional neural network is utilized, leveraging the Xception network optimized by an innovative algorithm known as the Improved Chaos African Vulture Optimization (ICAVO) algorithm. The ICAVO algorithm enhances the Xception network's performance by efficiently exploring the solution space and avoiding local optima. The results of this study demonstrate a significant enhancement in recognition accuracy, with the Xception/ICAVO model achieving remarkable recognition rates for both solid and striped balls. This paves the way for the development of more sophisticated and efficient billiards robots. The implications of this research extend beyond 8-ball billiards, highlighting the potential for advanced robotic vision systems in various applications. The successful integration of color image processing, deep learning, and optimization algorithms shows the effectiveness of the proposed methodology. This research has far-reaching implications that go beyond just billiards. The cutting-edge robotic vision technology can be utilized for detecting and tracking objects in different sectors, transforming industrial automation and surveillance setups. By combining color image processing, deep learning, and optimization algorithms, the system proves its effectiveness and flexibility. The innovative approach sets the stage for creating advanced and productive robotic vision systems in various industries.

Path Planning for Autonomous Mobile Robot Using Intelligent Algorithms

Machine learning technologies are being integrated into robotic systems faster to enhance their efficacy and adaptability in dynamic environments. The primary goal of this research was to propose a method to develop an Autonomous Mobile Robot (AMR) that integrates Simultaneous Localization and Mapping (SLAM), odometry, and artificial vision based on deep learning (DL). All are executed on a high-performance Jetson Nano embedded system, specifically emphasizing SLAM-based obstacle avoidance and path planning using the Adaptive Monte Carlo Localization (AMCL) algorithm. Two Convolutional Neural Networks (CNNs) were selected due to their proven effectiveness in image and pattern recognition tasks. The ResNet18 and YOLOv3 algorithms facilitate scene perception, enabling the robot to interpret its environment effectively. Both algorithms were implemented for real-time object detection, identifying and classifying objects within the robot’s environment. These algorithms were selected to evaluate their performance metrics, which are critical for real-time applications. A comparative analysis of the proposed DL models focused on enhancing vision systems for autonomous mobile robots. Several simulations and real-world trials were conducted to evaluate the performance and adaptability of these models in navigating complex environments. The proposed vision system with CNN ResNet18 achieved an average accuracy of 98.5%, a precision of 96.91%, a recall of 97%, and an F1-score of 98.5%. However, the YOLOv3 model achieved an average accuracy of 96%, a precision of 96.2%, a recall of 96%, and an F1-score of 95.99%. These results underscore the effectiveness of the proposed intelligent algorithms, robust embedded hardware, and sensors in robotic applications. This study proves that advanced DL algorithms work well in robots and could be used in many fields, such as transportation and assembly. As a consequence of the findings, intelligent systems could be implemented more widely in the operation and development of AMRs.

Target location method of intelligent deicing robot based on nonlinear auto disturbance rejection neural network

A visual navigation scheme for intermittent walking deicing robots was designed to address the issues of multiple iterations, long computation time, and poor real-time performance in the nonlinear optimization of the original SLAM system. A computer image acquisition unit, a computer image processing module, and a visual navigation parameter extraction algorithm were also designed. Implemented a visual navigation system for deicing robots based on image processing. It can meet the navigation requirements of the deicing robot. Simulation and experiments have shown that the method proposed in this paper can quickly and accurately identify abnormal point clouds. The visual servo control scheme constructed in this paper is aimed at robot task operations with unknown system calibration and target depth information. By calibrating the robot vision system, the conversion between camera pixel coordinates and robot base coordinates is achieved, with a transmission frame rate of 53.65 per second; The maximum error in positioning accuracy in space is 10.6 mm. The feature trajectory of visual space is smooth and stable within the camera's field of view, and the end movement of Cartesian space robots is stable without rebound, resulting in high grasping and positioning accuracy.

Kinematic and Joint Compliance Modeling Method to Improve Position Accuracy of a Robotic Vision System.

In the field of robotic automation, achieving high position accuracy in robotic vision systems (RVSs) is a pivotal challenge that directly impacts the efficiency and effectiveness of industrial applications. This study introduces a comprehensive modeling approach that integrates kinematic and joint compliance factors to significantly enhance the position accuracy of a system. In the first place, we develop a unified kinematic model that effectively reduces the complexity and error accumulation associated with the calibration of robotic systems. At the heart of our approach is the formulation of a joint compliance model that meticulously accounts for the intricacies of the joint connector, the external load, and the self-weight of robotic links. By employing a novel 3D rotary laser sensor for precise error measurement and model calibration, our method offers a streamlined and efficient solution for the accurate integration of vision systems into robotic operations. The efficacy of our proposed models is validated through experiments conducted on a FANUC LR Mate 200iD robot, showcasing notable improvements in the position accuracy of robotic vision system. Our findings contribute a framework for the calibration and error compensation of RVS, holding significant potential for advancements in automated tasks requiring high precision.

Rover with Enhanced Manipulation and Vision Capabilities

Abstract: From planetary exploration to terrestrial uses, rovers are essential in many sectors. This research aims to improve the adaptability and efficacy of the rover in a variety of contexts by presenting a revolutionary design that is outfitted with sophisticated manipulation and vision capabilities. Modern robotic grippers, arms, and vision systems are all integrated into the system architecture to provide a holistic solution for challenging jobs. The creation, integration, and coordination of these upgraded capabilities are described in depth in this work, with experimental data demonstrating improved performance and providing support. Case studies and real-world applications show the potential effect of the proposed rover, and talks about future work and problems offer ideas for additional improvements in rover technology. This work opens up new possibilities for improved capabilities in difficult circumstances and ads to the field of robotic exploration and manipulation

Low-cost parallel delta robot for a pick-and-place application with the support of the vision system

The main topic of this paper is how to develop pick and place delta robot with high efficiency mechanical parts and prosaically assemble it, additionally built electrical wiring diagram all of this supported by vision computer system. The system has a low-cost part and solves many problems to build this robot. Electrical section has two main components master and slave the master is raspberry pi and slave is Arduino, master is used to vision detect object and Arduino is used to invers kinematics equation, move motors to target position, control in hydraulic system to pull the object with suction cup. The process of vison system is passing through some steps, detect the object by color range, contouring box then send the real position to an Arduino. The experimental process proved the effectiveness and accuracy of the delta robot vision system in the sorting process from conveyor system. The target of system that function is applied with a low cost.

Investigation of Multi-Stage Visual Servoing in the context of autonomous assembly

This paper introduces a novel strategy, Multi-Stage Based Visual Servoing (MSBVS), to tackle the significant challenge of planning complex trajectories in robotic systems without hand–eye calibration, particularly in some assembly tasks. The MSBVS strategy decomposes a complete visual servoing process into three distinct phases, each characterized by unique functionalities. By assigning specific image features and control objectives to each phase, the strategy indirectly facilitates the planning and tracking complex trajectories. Furthermore, a novel image feature, the vanishing angle, is introduced, enhancing the precision in representing an object’s spatial position. Subsequently, a robotic vision system is established, and real-time simulations and experiments are performed to verify the efficacy of the MSBVS strategy. The results show that MSBVS efficiently plans and executes complex trajectories in robotic assembly tasks.

Occluded apples orientation estimator based on deep learning model for robotic harvesting

Apple harvesting robot is in rapid development in the recent years due to the shortage of manual labor. Robotics vision system has been one of the main focus in the development to achieve successful grasping. Information such as the apple orientation is valuable for developing more effective fruit detachment strategies. Occlusions in the orchard environment and the lack of distinct features in an apple has made it challenging to identify the apple orientation. Our study presents a novel approach that aims to estimate the orientation of fruits by projecting 2D information onto a 3D space. This method specifically targets the field of robotic harvesting and leverages multiple well-established and reliable fundamental methods currently available that includes keypoint detection neural network and circle detection based on extracted line segments of occluded apple. Based on the extracted information, a unit vector that represents orientation of the apple can be computed and transformed according to the relative position between the apple and the camera. The proposed method is evaluated using a publicly available 3D point cloud data of 11 apple trees from an orchard farm that was additionally labeled manually with apple orientations. To verify the evaluation results, the proposed method is tested by integrating it into an apple harvesting robot to perform inspection. Results shows that the median angular error in the orchard setting is 17.6° whilst in the lab setting is 14.6°. The experimental results show that the performance of the proposed method is comparable to existing research when under heavy occlusion and thus it is suitable for harvesting robot operating in the orchard farm.

Panoramic visual system for spherical mobile robots

Abstract Aimed at the challenges of wide-angle mobile robot visual perception for diverse field applications, we present the spherical robot visual system that uses a 360° field of view (FOV) for realizing real-time object detection. The spherical robot image acquisition system model is developed with optimal parameters, including camera spacing, camera axis angle, and the distance of the target image plane. Two 180 $^{\circ}$ -wide panoramic FOVs, front and rear view, are formed using four on-board cameras. The speed of the SURF algorithm is increased for feature extraction and matching. For seamless fusion of the images, an improved fade-in and fade-out algorithm is used, which not only improves the seam quality but also improves object detection performance. The speed of the dynamic image stitching is significantly enhanced by using a cache-based sequential image fusion method. On top of the acquired panoramic wide FOVs, the YOLO algorithm is used for real-time object detection. The panoramic visual system for the spherical robot is then tested in real time, which outputs panoramic views of the scene at an average frame rate of 21.69 fps and panoramic views with object detection at an average of 15.39 fps.