Visual Servo Control System Research Articles

Crop Recognition Method Based On End-effector

With the increasing global population, the demand for agriculture is also on the rise. The crucial stages of agricultural production, namely fruit identification and picking, play a vital role in enhancing product quality and minimizing losses. Traditional manual processing methods, although time-tested, are not only inefficient but also challenging to maintain consistency, making them inadequate to meet the large-scale requirements of modern agricultural production. Consequently, the integration of automation technology has become a necessity. For agricultural robot the machine vision system often need to work in two typical environments, the field environment and the orchard environment. Depending on the varying objectives of operation in diverse environments, crop robots require the ability to rapidly identify fruits within images featuring significant color disparities between crops and two-dimensional field backgrounds. Consequently, a visual servo control system is being investigated. A novel camera attitude search method that employs active visual servo technology to minimize occlusions during orchard search is proposed. The recognition function of the end-effector is exceedingly crucial. The precision of the end effector's identification capabilities directly influences the success rate of automated operations. This is particularly evident in fruit picking, sorting, and other tasks, where the diverse shapes, maturity levels, and colors of fruits present significant challenges to the robotic arm's end effector. The rapid advancement of deep learning technology, however, offers a novel solution for the recognition of fruits by the robotic arm's end effector. By emulating the human visual system, deep learning models can extract the feature representation of fruits from vast amounts of data, enabling accurate identification of fruits in various conditions.

In-Depth Evaluation of Automated Fruit Harvesting in Unstructured Environment for Improved Robot Design

Using modern machines like robots comes with its set of challenges when encountered with unstructured scenarios like occlusion, shadows, poor illumination, and other environmental factors. Hence, it is essential to consider these factors while designing harvesting robots. Fruit harvesting robots are modern automatic machines that have the ability to improve productivity and replace labor for repetitive and laborious harvesting tasks. Therefore, the aim of this paper is to design an improved orange-harvesting robot for a real-time unstructured environment of orchards, mainly focusing on improved efficiency in occlusion and varying illumination. The article distinguishes itself with not only an efficient structural design but also the use of an enhanced convolutional neural network, methodologically designed and fine-tuned on a dataset tailored for oranges integrated with position visual servoing control system. Enhanced motion planning uses an improved rapidly exploring random tree star algorithm that ensures the optimized path for every robot activity. Moreover, the proposed machine design is rigorously tested to validate the performance of the fruit harvesting robot. The unique aspect of this paper is the in-depth evaluation of robots to test five areas of performance that include not only the accurate detection of the fruit, time of fruit picking, and success rate of fruit picking, but also the damage rate of fruit picked as well as the consistency rate of the robot picking in varying illumination and occlusion. The results are then analyzed and compared with the performance of a previous design of fruit harvesting robot. The study ensures improved results in most aspects of the design for performance in an unstructured environment.

Position-Based Visual Servo Control of Dual Robotic Arms With Unknown Kinematic Models: A Cerebellum- Inspired Approach

The inverse kinematics problem of robotic arms can be solved effectively based on the kinematic model. However, the control of robotic arms with unknown kinematic models remains challenging. This article proposes a position-based visual servo (PBVS) control system for dual robotic arms with unknown kinematic models. First, a unified PBVS control method based on dual gradient neural dynamic (DGND) models is developed to achieve the position and orientation control of a single robotic arm with unknown kinematic model. Next, a cerebellum-inspired DGND (CIDGND) control system is devised to achieve higher accuracy. More importantly, a control system for dual robotic arms is proposed based on the CIDGND method to expand the working space of the primary robotic arm, by autonomously adjusting the camera pose with another robotic arm. Finally, simulations and experiments are performed to verify the efficacy of the proposed control systems. The results validate that the DGND method can achieve the PBVS control of robotic arms in the absence of kinematic models and the CIDGND method is able to reduce the tracking error significantly. Comparative results reveal that the CIDGND method is effectively and robustly better than existing methods.

Predictor-Based Extended State Observer for Disturbance Rejection Control of Multirate Systems With Measurement Delay

This article investigates the multirate disturbance rejection control problem for linear control systems with mismatched disturbance and measurement delay using predictor-based extended state observer. A new extended state observer together with output predictor is first designed to obtain the estimation values of system state and mismatched disturbance, where output predictor is used to compensate the influences of measurement delay and sampling of output. To attenuate the undesirable influence of mismatched disturbance, we then design a new sampled-data robust controller with disturbance compensation, and the updating rate of the proposed controller is allowed to be different from that of the sensor. Thanks to prediction and disturbance/uncertainty estimation and attenuation techniques, the disturbance rejection property of the resultant closed-loop control systems is enhanced despite the multirate and measurement delay. Some sufficient conditions are presented to ensure the stability property of the resultant control systems. We finally consider the application of the visual servoing control system for an inertially stabilized platform, and the experiment results verify superiorities of the predictor-based disturbance rejection control method proposed in the article.

An image-based visual servo system to operate a flexible wire

PurposeWith the rapid development of the 3C industry, the problem of automated operation of 3C wire is becoming increasingly prominent. However, the 3C wire has high flexibility, and its deformation is difficult to model and control. How to realize the automation operation of flexible wire in 3C products is still an important issue that restricts the development of the 3C industry. Therefore, this paper designs a system that aims to improve the automation level of the 3C industry.Design/methodology/approachThis paper designed a visual servo control system. Based on the perception of the flexible wire, a Jacobi matrix is used to relate the deformation of the wire to the action of the robot end; by building and optimizing the Jacobi matrix, the robot can control the flexible wire.FindingsBy using the visual servo control system, the shape and deformation of the flexible wire are perceived, and based on this, the robot can control the deformation of the flexible wire well. The experimental environment was built to evaluate the accuracy and stability of the system for controlling the deformation of the flexible wire.Originality/valueAn image-based visual servo system is proposed to operate the flexible wire, including the vision system, visual controller and joint velocity controller. It is a scheme suitable for flexible wire operation, which has helped to automate flexible wire-related industries. Its core is to correlate the motion of the robot end with the deformation of the flexible wire through the Jacobian matrix.

Fuzzy Gain-Scheduling Based Fault Tolerant Visual Servo Control of Quadrotors

When military and civil missions such as transportation increase, fault tolerant control of unmanned aerial vehicles will be an obligation. Although onboard sensors provide information about the status of a quadrotor, the camera is not included in the list. In this study, visual servo control of quadrotors as a popular method for motion control is addressed. we address a visual servo control system for quadrotors as a popular method for motion control. The feature motions in the image plane are analyzed to reveal the relation between the actuator faults and these motions. Four AI fault approximators, a neural network, an extreme learning machine, a linear support vector machine, and a long short-term memory are used to approximate actuator faults of a quadrotor while using feature inputs. The results are convincing and the approximation results are used by a fuzzy logic unit to provide gain-scheduling based fault tolerant control. The proposed system shows sufficient results as a visual servo system for fixed and moving feature targets while providing fault tolerance.

Autonomous 6-DOF Manipulator Operation for Moving Target by a Capture and Placement Control System.

The robot control technology combined with a machine vision system provides a feasible method for the autonomous operation of moving target. However, designing an effective visual servo control system is a great challenge. For the autonomous operation of the objects moving on the pipeline, this article is dedicated to developing a capture and placement control system for the six degrees of freedom (6-DOF) manipulator equipped with an eye-in-hand camera. Firstly, a path planning strategy of online capture and offline placement is proposed for real-time capture and efficient placement. Subsequently, to achieve the fast, stable, and robust capture for a moving target, a position-based visual servo (PBVS) controller is developed by combining estimated velocity feedforward and refined PID control. Feedforward control is designed using the estimated velocity by a proposed motion estimation method for high response speed. PID control is refined by dead zone constraint to reduce the manipulator’s jitter caused by the frequent adjustment of manipulator control system. Besides, the proportional, integral, and differential coefficients of PID controller are adaptively tuned by fuzzy control to reject the noise, disturbance, and dynamic variation in the capture process. Finally, validation experiments are performed on the constructed ROS–Gazebo simulation platform, demonstrating the effectiveness of the developed control system.

Research on Design and Implementation of Computer 3D Table Tennis Simulation Animation

Table tennis is China’s national ball, which leads to education and teaching is particularly important. In the daily teaching of table tennis technical movements, it is difficult for coaches to achieve one-to-one training mode. At the same time, due to the different foundation of students, it is difficult for coaches to teach students in accordance with their aptitude. At the same time, based on the objective factors such as the angle and position of table tennis, the coach’s judgment may be biased, which leads to the realization of 3D table tennis simulation animation based on computer. Through 3D table tennis simulation animation, we can not only improve the shortcomings of traditional table tennis teaching, but also make table tennis teaching more modern. Therefore, this paper first analyzes the movement principle of table tennis. Then, this paper designs the visual servo control system, which can better watch the entity training. Finally, this paper proposes the implementation method.

A Visual Servo Control Method for Tomato Cluster-Picking Manipulators Based on a T-S Fuzzy Neural Network

HighlightsA T-S fuzzy neural network was applied to the visual servo control system of a tomato picking manipulator.The T-S fuzzy neural network structure was designed, and collected data were used to train the neural network model.A visual servo control system for the picking manipulator based on the neural network was designed and tested.The T-S fuzzy neural network was superior to a BP neural network in visual servo control of the picking manipulator.Abstract. To reduce the computational load of image Jacobian matrix estimation and to avoid the appearance of singularity of a Jacobian matrix in the visual servo control of a picking manipulator, a T-S fuzzy neural network algorithm is proposed to replace the image Jacobian matrix. This better fits the hand-eye relationship by combining the knowledge structure of fuzzy reasoning with the self-learning ability of a neural network. The T-S fuzzy neural network was trained and tested by collecting the variation data of image features and joint angles; after training, the T-S fuzzy neural network was used to predict the joint angles of the picking manipulator. Simulation results show that the square sum of training errors and testing errors were 0.017 and 0.032, respectively, after training the T-S fuzzy neural network. A T-S fuzzy neural network controller was applied to the visual servo system of the picking robot, and the test results show that the average difference between the end-effector and the ultimate target location of the visual servo system based on the T-S fuzzy neural network controller was 0.0037 m, which was 79.44% less than that of the visual servo system based on a BP neural network. The final average error of image features was between 0.52 and 3.25 pixels, which was 74.932% less than that of the visual servo system based on the BP neural network. Keywords: Picking manipulator, Tomato clusters, T-S fuzzy neural network, Visual servoing.

Research on Robot Vision Servo Based on image big data

With the integration and development of information technology and control theory, industrial robot has become an indispensable automation equipment in modern industrial production, improving product quality and work efficiency. However, there is still a need to improve robot technology in terms of intelligence and flexibility. Robot vision servo control technology uses image sensors to obtain the information of the external field of vision, which can improve its environmental perception and response ability to cope with different production work. The important control method of robot system is visual servo control. Because of its good performance, wide application and convenience, visual servo control technology has become one of the hot spots in the field of mechatronics. In this paper, the problem of visual feedback and noise processing in the vision system is described. The six degree of freedom pose correction of the three focus tensor is made use of the big image data, and some tensor elements are selected to define the visual feedback, which effectively avoids the display estimation of the camera pose. Based on the extracted tensor features, an adaptive controller is designed to drive the camera to the desired attitude. The visual servo control system improves the real-time performance of big data image processing and reduces the calibration error, achieving the expected purpose. Finally, the future research direction is prospected, which is expected to become an important support technology for robot application in dynamic and variable environment.

An Automated AFM Tip-Replacement System for In Situ Tip-Replacement

The increase in applications of the atomic force microscope (AFM) in industrial metrology has contributed to its increased automation. However, automatic replacement of the AFM tip without constraining the instrument's capabilities is still an important challenge. Here, in this article, we propose a system for automated replacement of the AFM tip that is compatible with all imaging modalities and requires minimal modification of a conventional AFM. The proposed strategy employs a wax microsphere as an adhesive for attaching a tip to a tip-less AFM probe and to subsequently detach it. A laser-based heating system and a visual servo control system, which includes an in-house developed out-of-plane motion measurement strategy, are employed for fast and accurate tip-replacement. The system is fabricated, calibrated, and automated tip-replacement is demonstrated with a tip positioning accuracy better than 0.3 μm and an average tip-replacement time of 10 s. Automated replacement of a deteriorated tip has been demonstrated in two specific cases, one in the context of repeated imaging, and another in the context of nanoindentation.

A Real-Time Image Encryption Method of Networked Inverted Pendulum Visual Servo Control System

In the real-time control of networked inverted pendulum visual servo control system (NIPVSCS), how to achieve safe and efficient image transmission is an important and challenging problem. Generally, the inverted pendulum images are directly transmitted via the network, while these images are not secure because they may be attacked by the hacker. To solve the problem, a fast image encryption method by combining image scaling and improved self-diffusion image encryption (ISDIE) algorithm is firstly proposed, which is employed to encrypt the captured images after they are scaled. The security performance of the proposed algorithm, the characteristics of different delays, and the control performance of the NIPVSCS are then analyzed. Finally, simulation and real-time control experiments confirm the feasibility and effectiveness of the proposed method.

A novel field box girder welding robot and realization of all-position welding process based on visual servoing

Abstract Box-type steel structure has been widely applied in the field of buildings and bridges because of its unique advantages. However, there are still many problems in field welding that need to be solved. In order to improve the level of automation and intellectualization in field welding of box girder, a new type of on-site all-position welding robot system is developed and designed in this paper. The convenience of on-site installation and debugging of the robot is realized by the scheme of permanent magnet adsorption and straight-arc combined track. The motion matching scheme of bilateral wheel-axle clamping makes the circumferential movement of the robot much smoother and more stable which makes the system with better overall force characteristics and load-carrying capacity. Then, a novel visual sensor based on combined laser structured lights is proposed and designed, which can realize several functions such as groove parameters detection, welding joint tracking and torch posture detection in field welding based on single detection image, so as to effectively improve the flexibility and intelligence of field welding robot. After that, based on the designed visual sensor and welding robot system, a visual servo-control system based on Visual Studio platform is established. Through the effective integration of detection image reading, image processing and calculation, image feature parameters transmission, motion control board call and feedback control signal output, the adjustment and control of welding torch's position and posture based on visual servo-control are realized. Furthermore, based on the kinematics calculation and the welding process requirements of the box girder, the welding process design is planned for the all-position welding of the box steel girder, which solved the technical problem of the rectangular fillet welding in the box steel butt welding (the welding torch position and posture transition control in right-angled corner of girth butt joint). Finally, all-position welding experiments of box steel girder were carried out. The results show that the welding robot based on visual servo-control can realize better control of weld formation in all-position welding of box steel girder.

Research on Visual Control System of Inverted Pendulum Based on Pixel Displacement

In inverted pendulum visual servo control system (IPVSCSs), how to achieve the fast and effective measurement of the state information of the inverted pendulum is an important task. In this paper, the characteristics of the inverted pendulum are considered and a series of image processing and positioning algorithm of the pendulum are proposed. Furthermore, the inverted pendulum control system model based on the pixel displacement is established and the LQR controller is designed. Finally, the feasibility and superiority of the proposed method are verified by real-time control experiments.

An Novel Selective Image Encryption Algorithm For Networked Visual Inverted Pendulum

Abstract The inverted pendulum image may be subject to various cyber attacks in the networked inverted pendulum visual servo control system (NIPVSCS), how to ensure the safe and efficient transmission of image data is a challenging issue. To solve this problem, this paper proposes a novel selective image encryption algorithm. First, since the encrypted inverted pendulum image will bring long computing time that makes the system performance worse, the image is scaled by using the bilinear difference method to initially reduce the amount of image data. Second, to further improve the computation speed to meet the real-time requirement of the NIPVSCS, Hough transform is employed to determine the swing rod of inverted pendulum image, and the encrypted area is set to further reduce the calculation complexity. Third, according to the key area, chaotic map is used to generate a small number of random rows and columns, which are cyclically shifted to break the strong correlation of adjacent pixels, then the XOR operation is used to mask the pixel values. Finally, simulation experiments confirm the feasibility and effectiveness of the proposed method.

Robust Visual Servoing Control for Ground Target Tracking of Quadrotors

In this brief, the robust position-based visual servoing control problem is addressed for a quadrotor with a down-facing monocular camera to perform ground target tracking tasks in outdoor environments. The object is to regulate the position of the quadrotor relative to the ground target by using a minimum sensor set, including an inertial measurement unit, an ultrasonic sensor, and a vision sensor. The robust compensators are designed in the visual servoing control system to enhance the robustness against nonlinearities, couplings, and uncertainties, including parameter uncertainties and external disturbances in the translational and rotational quadrotor dynamics. Theoretical analysis and experimental results demonstrate the effectiveness of the proposed robust visual servoing control method.

Real-Time H∞ Control of Networked Inverted Pendulum Visual Servo Systems.

Aiming at the challenges of networked visual servo control systems, which rarely consider network communication duration and image processing computational cost simultaneously, we here propose a novel platform for networked inverted pendulum visual servo control using H∞ analysis. Unlike most of the existing methods that usually ignore computational costs involved in measuring, actuating, and controlling, we design a novel event-triggered sampling mechanism that applies a new closed-loop strategy to dealing with networked inverted pendulum visual servo systems of multiple time-varying delays and computational errors. Using the Lyapunov stability theory, we prove that the proposed system can achieve stability whilst compromising image-induced computational and network-induced delays and system performance. In the meantime, we use H∞ disturbance attenuation level γ for evaluating the computational errors, whereas the corresponding H∞ controller is implemented. Finally, simulation analysis and experimental results demonstrate the proposed system performance in reducing computational errors whilst maintaining system efficiency and robustness.

Robust Kalman filtering with long short-term memory for image-based visual servo control

Visual servo control systems based on Kalman filter (KF) is susceptible to noise interference, the initialization of the Jacobi matrix is difficult, and the observation value of the Jacobian matrix is not accurate. In order to address these problems, we proposed a robust KF algorithm with long short-term memory (LSTM) for an image-based visual servo control system and applied the system to an uncalibrated image-based visual servo (IBVS) control system to estimate the filtering gain error, state estimation error, and the observation error, which were then used for online training in LSTM. The visual servo control system controls the motion of the manipulator, and simultaneously updates the LSTM network. Therefore, the Jacobian matrix obtained using LSTM was employed to estimate the state volume of the robust KF, which constitutes a circulatory system, and the complementary effect was realized. The method was applied to a six-degrees-of-freedom manipulator of the eye-in-hand model to conduct experiments. The simulation results indicate that the proposed visual servo control system has strong anti-noise interference capability. Furthermore, it facilitates Jacobian matrix initialization and has high observation accuracy for the Jacobian matrix.

Adaptive Image-Based Visual Servoing With Temporary Loss of the Visual Signal

Image-based visual servoing (IBVS) can reach a desired position for a relatively stationary target using continuous visual feedback. Proper feature extraction and appropriate servoing control laws are essential to performance for IBVS. IBVS control can be interrupted or interfered abruptly if no features are extracted when the observed object is occluded. To address the problem of missing feature points in current images during a visual navigation task, a homography method that uses a priori visual information is proposed to predict all of the missing feature points and to ensure the execution of IBVS. The mixture parameter for the image Jacobian matrix can also affect the control of IBVS. The settings for the mixture parameter are heuristic so there is no a systematic approach for most IBVS applications. An adaptive control approach is proposed to determine the mixture parameter. The proposed method uses a reinforcement learning (RL) method to adaptively adjust the mixture parameter during the robot movement, which allows more efficient control than a constant parameter. A logarithmic interval state-space partition for RL is used to ensure efficient learning. The integrated visual servoing control system is validated by several experiments that involve wheeled mobile robots reaching a target with a desired configuration. The results for simulation and experiment demonstrate that the proposed method has a faster convergence rate than other methods.

A learning approach to image-based visual servoing with a bagging method of velocity calculations

Visual servoing allows accurate control of positioning and motion relative to a stationary or moving target using vision and is the subject of many studies. Most servoing gains for image-based visual servoing methods are selected heuristically or empirically so accuracy is affected. This study uses reinforcement learning to adaptively tune the proportional servoing gain over sequences of image based visual servoing, instead of using a constant gain. The system is used to control hovering and tracking for a stationary or slowly moving target. An image Jacobian matrix with four dimensions is constructed because a quad-rotor drone features under-actuated dynamics. The Moore-Penrose pseudo inverse method is usually used to calculate the inverse image Jacobian matrix, but this study uses a bagging approach to calculate the inverse kinematics. The desired velocity is obtained from time-varying image errors, which gives greater robustness. An adaptive method to calculate the servoing gain method is proposed, whereby the selection of an appropriate servo gain over time is regarded as a reinforcement learning problem. The proposed visual servoing control system is implemented and tested experimentally using a quad-rotor drone system. The experimental results demonstrate that the proposed method is more robust and converges faster than other methods.