Monocular Vision System Research Articles

EMG model calibration and trajectory tracking of rehabilitation exoskeleton by asynchronous tele-training

ABSTRACT This study presents a novel kinematic tracking model, designed for a networked exoskeleton system that is asynchronously taught by a remote therapist. The therapist’s rehabilitation exercises are quantitatively assessed using a monocular vision system. The resultant metrics are then transmitted asynchronously over the network to patients equipped with exoskeletons. The exoskeleton utilises these metrics as reference paths for exercises, complemented by electromyography (EMG) feedback. This work introduces a calibration approach aimed at estimating angular positions by utilising EMG observations. The calibration model establishes real-time correlations between polynomial reference positions. We further explore redundant kinematics, incorporating an EMG observer for linear, time-variant rehabilitation tracking control. Our methodology is validated using vision-based metric data and experimental EMG measurements including shoulder flexion, elbow flexion, and rowing-like movements. Computer simulations demonstrate the system’s ability to reliably, robustly, and effectively follow desired trajectories. This research offers a promising approach for remote personalized rehabilitation.

Monocular visual obstacle avoidance method for autonomous vehicles based on YOLOv5 in multi lane scenes

This study explores a more effective obstacle avoidance method for autonomous driving based on the monocular vision system of YOLOv5. The study utilizes the YOLOv5 model to detect obstacles and road signs in the environment in real-time, including vehicles, pedestrians, traffic signals, etc., identifying objects of different sizes and angles, providing accurate obstacle avoidance decision inputs. Then, we use a path planning algorithm based on deep reinforcement learning to combine the detected obstacle information with the current state of the vehicle, dynamically generating safe and efficient driving paths. In order to further improve the obstacle avoidance effect, a monocular visual autonomous driving obstacle avoidance aggregation network was introduced in the study, and the MMA (Multi lane monocular visual autonomous driving) obstacle avoidance method was established. The future movement trend of obstacles was predicted by setting an improved monocular obstacle avoidance loss function to sense historical data and the environment. The experimental results show that MMA has significantly enhanced obstacle avoidance and driving efficiency. The accuracy of the MMA obstacle avoidance method fluctuates between 78.76 % and 88.26 %, showing the best overall performance. The accuracy also shows an upward trend with the increase of the data sample size.

Research on Monocular Depth Sensing Method Based on Liquid Zoom Imaging

Monocular stereo vision has excellent application prospects in the field of microrobots. On the basis of the geometric model of bifocal imaging, this paper proposes a monocular depth perception method by liquid zoom imaging. Firstly, the configuration of a monocular liquid vision system for depth measurement is presented, and the working mechanism of the system is analyzed through theoretical derivation. Then, to eliminate the influence of optical axis drift induced by the liquid gravity factor on the measurement results, the target image area is used as the calculation feature instead of the image vector length. A target area calculation method based on chain code classification and strip segmentation is proposed. Furthermore, in response to the fluctuation problem of liquid lens focal power caused by factors such as temperature and object distance, a dynamic focal length model of the liquid zoom imaging system is constructed after precise calibration of the focal power function. Finally, a testing experiment is designed to validate the proposed method. The experimental results show that the average error of depth perception methods is 4.30%, and its measurement time is only on the millisecond scale. Meanwhile, the proposed method has good generalization performance.

MONO-HYDRA: REAL-TIME 3D SCENE GRAPH CONSTRUCTION FROM MONOCULAR CAMERA INPUT WITH IMU

Abstract. The ability of robots to autonomously navigate through 3D environments depends on their comprehension of spatial concepts, ranging from low-level geometry to high-level semantics, such as objects, places, and buildings. To enable such comprehension, 3D scene graphs have emerged as a robust tool for representing the environment as a layered graph of concepts and their relationships. However, building these representations using monocular vision systems in real-time remains a difficult task that has not been explored in depth.This paper puts forth a real-time spatial perception system Mono-Hydra, combining a monocular camera and an IMU sensor setup, focusing on indoor scenarios. However, the proposed approach is adaptable to outdoor applications, offering flexibility in its potential uses. The system employs a suite of deep learning algorithms to derive depth and semantics. It uses a robocentric visual-inertial odometry (VIO) algorithm based on square-root information, thereby ensuring consistent visual odometry with an IMU and a monocular camera. This system achieves sub-20 cm error in real-time processing at 15 fps, enabling real-time 3D scene graph construction using a laptop GPU (NVIDIA 3080). This enhances decision-making efficiency and effectiveness in simple camera setups, augmenting robotic system agility. We make Mono-Hydra publicly available at: https://github.com/UAV-Centre-ITC/Mono_Hydra.

Autonomous relative navigation using stereo-vision in a dual-agent system for proximity operations in a lunar orbit

This paper presents a novel autonomous relative navigation architecture for inspector spacecraft in proximity operations to a known structure in a lunar orbit. Autonomous on-orbit servicing and assembly of large space structures are at the forefront of research for advancing space exploration beyond low-earth orbit. Communication delays with ground stations and limited computational power located on the inspector spacecraft make current relative navigation methods unfeasible and in turn, make it difficult to maintain an accurate estimate of the relative position and orientation of the inspector spacecraft with respect to the structure. The proposed method utilizes the camera measurements from two identical inspector spacecraft, each equipped with a single camera, to estimate the relative position and orientation of the structure. This shared measurement strategy overcomes the deficiencies in monocular vision-based relative navigation techniques, such as the range ambiguities, as is proven in the simulated results of this paper. The proposed stereo-vision system in this work allows for a deterministic solution of the relative position and orientation from the measurements of each inspector spacecraft to the structure. The stereo-vision system was tested with a Multiplicative Extended Kalman Filter (MEKF) for each spacecraft to estimate the relative states of the structure while maintaining a fixed relative position to the structure. The performance of the proposed stereo-vision system was evaluated in terms of accuracy and reliability of the estimated relative position and orientation of two inspector spacecraft to the structure over 100 Monte Carlo simulations for three case studies. The results showed that the stereo-vision system significantly outperforms the monocular vision system even when fewer points on the structure are visible, as is the case for proximity operations. Additionally, the stereo-vision system is consistently more accurate for the relative velocity, attitude, and angular velocity states regardless of how many points are visible. The proposed stereo-vision relative navigation system has the potential to be applied autonomously in proximity operations of spacecraft for various space missions, such as rendezvous and docking, inspection, and on-orbit servicing.

A localization method of manipulator towards achieving more precision control

AbstractThe monocular vision system is a crucial branch of machine vision research widely used in multiple industries as a research hotspot in the field of vision. Although the monocular vision system is of simple structure and cost‐effectiveness, its positioning accuracy is insufficient in some industries. This article researched the robot arm positioning method via monocular vision. First, we built a vision system model and designed the style of a cooperative target for target positioning. Second, a target feature screening method based on conditions is composed for the existence of interference. Furthermore, combining the principle of pose estimation on the PNP (Perspective‐n‐Point) problem with the results of the visual system calibration to realize the positioning of the target. Finally, complete the construction of the experimental platform and design accuracy evaluation experiments and positioning experiments. The experimental results show that the location measurement error range of the system in this article is below 4 mm, and the measurement error of the rotation angle is below 2. The system can adapt to the requirements of general industrial use.

A low-cost monocular vision system for robot calibration

The inverse kinematic model uses robot design parameters: ideal link lengths and mounting angles. In practice, these values hardly coincide with the design values due to manufacturing and assembly processes or continuous use of the robot. In order to reduce this geometric error, it is necessary to calibrate the robot to update the geometric model and reduce the resulting error of the robot end-effector. In this work, we propose a methodology based on a vision system to calibrate the robot's geometric parameters and minimize the error between the robot’s end-effector theoretical and real trajectory. This way, variations are introduced to the geometric parameters that generate errors between the robot's desired position and the position developed. The results show up a reduction of the average position error of 54.6%.

Spatio-Temporal Fusion of LiDAR and Camera Data for Omnidirectional Depth Perception

Object recognition and depth perception are two tightly coupled tasks that are indispensable for situational awareness. Most autonomous systems are able to perform these tasks by processing and integrating data streaming from a variety of sensors. The multiple hardware and sophisticated software architectures required to operate these systems makes them expensive to scale and operate. This paper implements a fast, monocular vision system that can be used for simultaneous object recognition and depth perception. We borrow from the architecture of a start-of-the-art object recognition system, YOLOv3, and extend its architecture by incorporating distances and modifying its loss functions and prediction vectors to enable it to multitask on both tasks. The vision system is trained on a large database acquired through the coupling of LiDAR measurements with complementary 360-degree camera to generate a high-fidelity labeled dataset. The performance of the multipurpose network is evaluated on a test dataset consisting of a total of 7,634 objects collected on a different road network. When compared with ground truth LiDAR data, the proposed network achieves a mean absolute percentage error rate of 11% on the passenger car within 10 m and a mean error rate of 7% or 9% on the truck within 10 m and beyond 10 m, respectively. It was also observed that adding a second task (depth perception) to the modeling network improved the accuracy of object detection by about 3%. The proposed multipurpose model can be used for the development of automated alert systems, traffic monitoring, and safety monitoring.

KCF-Based Identification Approach for Vibration Displacement of Double-Column Bents under Various Earthquakes

Vibration displacements are one of the most significant indicators in the health monitoring and condition assessment of bridges in the life cycle. The traditional monitoring means, such as contact sensors, have relatively high-cost and limited points for displacement measurement of bridges. This paper proposes a low-cost and non-contact monocular vision system based on the KCF algorithm to accurately and timely identify the vibration displacement of bridges. A conversion method associated with a scale ratio was established to cope with the loss of depth information in images when a monocular camera is used to monitor multiple targets in different depths of the field. A series of shaking table tests on a two-column pier with energy dissipation beams were conducted to verify the feasibility, accuracy, effectiveness, and robustness of the KCF-based identification approach. The results showed that the vibration displacements of the column identified by the monocular vision system based on the KCF algorithm are almost consistent with the measurement results obtained by the laser displacement sensors. The peak displacement discrepancies between both measurement methods are within 6% for all cases with different shaking amplitudes and earthquake waves. The RMSE of the displacement histories between both measurement methods is very low. The corresponding frequency spectra contents identified by the monocular vision system based on the KCF algorithm match well with the measurement counterparts recorded from the laser displacement sensors.

Fusing few-shot learning and lightweight deep network method for detecting workpiece pose based on monocular vision systems

With the popularization of workpiece grabbing and sorting applications, the measurement accuracy and speed of workpiece pose become the key issues. Therefore, a lightweight workpiece pose measurement algorithm based on the YOLO framework is proposed. In the study, the backbone network of YOLOv4-Tiny is lightened by the MobileNetV3 module. Then, the rotation anchor and loss function are introduced into the network to measure the pose. Additionally, to improve the adaptability of the algorithm to novel workpieces, a two-stage fine-tuning strategy is integrated into the pose measurement network to realize few-shot learning. Compared with the general object detection network, the proposed algorithm can achieve high speed and precision pose measurement and has good adaptability to novel workpieces. Experimental results show that the measurement speed of the proposed method is eight times faster than baseline Rotation-YOLOv4-Tiny, the position error is 0.42 × 10−3 m, and the angle error is 0.016 rad.

Measurement of Six-Degree-of-Freedom Absolute Postures Using a Phase-Encoded Pattern Target and a Monocular Vision System

Measurement of Six-Degree-of-Freedom Absolute Postures Using a Phase-Encoded Pattern Target and a Monocular Vision System

A novel hand-eye calibration method using double-layer optimization and outlier sample screening for monocular vision robots

In monocular vision robot systems, the hand-eye calibration approach is crucial for ensuring operational accuracy. A double-layer (Levenberg–Marquardt ([LM] method) optimization (DLMO) method with outlier points screening is proposed to reduce the influence of random errors in robot vision systems and improve the calibration accuracy of the robot hand-eye matrix. First, the equation of the hand-eye matrix is established, and the initial value of the hand-eye matrix is solved by the linear least square method. Second, the Euler angle transformation is applied to the rotation matrix part to ensure its orthogonality. Next, an optimization model of the hand-eye matrix is constructed, and the traditional LM optimization method is used to optimize the initial hand-eye matrix for the first time. Finally, the optimization model of the hand-eye matrix is modified, and the LM optimization method with the outlier sample points screening is applied to optimize the hand-eye matrix for the second time. Hand-eye calibration tests are conducted on an industrial robot equipped with a monocular vision system using the proposed method. Experimental results demonstrate that the average position error of the calibration results obtained by the proposed DLMO method is 0.22 mm, which is superior to the traditional hand-eye calibration method and meets the working requirements of the vision robot in the industrial field.

Industrial Robot Assembly Line Design Using Machine Vision

In order to further improve the functional requirements and performance indicators of the industrial robot assembly system and more accurately realize the measurement and recognition of the target position of the assembly line by the vision system, this article constructs a robot assembly line system based on obstacle detection and robot arm obstacle path planning based on machine vision technology and further improves the intelligence and accuracy of the assembly line system through the design and optimization of the system software module. Through the experimental verification of the positioning error based on the eye-to-hand binocular vision system and eye-in-hand monocular vision system, the system proposed in this article meets the design accuracy requirements of less than 0.1 mm in x/y direction and less than 1 mm in depth direction and verifies the feasibility and high accuracy of the system.

Hand Tracking and Gesture Recognition by Multiple Contactless Sensors: A Survey

Hand tracking and gesture recognition are fundamental in a multitude of applications. Various sensors have been used for this purpose, however, all monocular vision systems face limitations caused by occlusions. Wearable equipment overcome said limitations, although deemed impractical in some cases. Using more than one sensor provides a way to overcome this problem, but necessitates more complicated designs. In this work, we aim to highlight contemporary methods used for hand tracking and gesture recognition by collecting publications of systems developed in the last decade, that employ contactless devices as RGB cameras, IR, and depth sensors, along with some preceding pillar works. Additionally, we briefly present common steps, techniques, and basic algorithms used during the process of developing modern hand tracking and gesture recognition systems and, finally, we derive the trend for the next future.

Sensing Algorithm to Estimate Slight Displacement and Posture Change of Target from Monocular Images.

Various types of displacement sensors, which measure position changes of object, have been developed depending on the type and shape of the object under measurement, measurement range of the amount of displacement, required accuracy, and application. We are developing a new type of displacement sensor that is image-based, capable of measuring changes in 6DOF (3D position and orientation) of an object simultaneously, and is compact and low-cost. This displacement sensor measures the 6DOF of an object using images obtained by a monocular vision system. To confirm the usefulness of the proposed method, experimental measurements were conducted using a simple and inexpensive optical system. In this experiment, we were able to accurately measure changes of about 0.25 mm in displacement and 0.1 deg in inclination of the object at a distance of a few centimeters, and thus confirming the usefulness of the proposed method.

Weighted voting ensemble method for predicting workpiece imaging dimensional deviation based on monocular vision systems

Due to the different heights and positions of all kinds of workpieces, the imaging dimension of workpieces in monocular vision systems exists in deviation. To reduce the deviation and improve the accuracy of dimensional measurement, a BRR-SVR-BPNN weighted voting ensemble learning deviation prediction algorithm is proposed. The proposed method combines the probability inference feature of Bayesian Ridge regression (BRR), the maximum interval penalty feature of support vector regression (SVR), and the nonlinear expression feature of the backpropagation neural network (BPNN). And an automatic image processing method based on Halcon is proposed to establish the datasets for the deviation prediction algorithm. The experimental results show that the proposed algorithm has well prediction effect, high robustness, and accuracy on deviation prediction. It can be easily applied to predict the imaging dimension deviation of various workpieces or other similar tasks to improve the accuracy of workpiece dimension measurement in the industrial field.

Monocular vision pose determination-based large rigid-body docking method

Large rigid-body automatic docking is a crucial but tough task in aerospace assembly. Traditional large rigid-body spacecraft assembly is achieved manually, which is limited by high labor-intensity, low assembly quality, and long assembly cycle. This study proposes the monocular vision pose determination-based large rigid-body automatic docking method (MVPDLD) in which targets with circular feature points are attached to the surfaces of both mobile and fixed rigid-bodies; Thus, the proposed monocular vision system can track the relative pose between them. The measured relative pose is converted into kinematic parameters for a lightweight five-degree-of-freedom pose adjustment mechanism (5-DOFPAM), which can be guided via monocular vision pose determination to adjust the mobile rigid-body’s pose and achieve accurate docking. The MVPDLD method was validated on real datasets. MVPDLD’s accuracy is similar to traditional methods but achieves accurate docking much quicker. It can replace traditional manual methods for the automatic docking of large rigid-bodies.

Motion compensation system for robotic based in situ bioprinting to balance patient physiological movements

The aim of this study is to design and develop a robotic system capable of compensating a patient's periodic movement, such as a beating heart, breath-induced thoracic cavity motion, and able to avoid collisions in case of sudden and unexpected motions, caused by pain, tremor, or other diseases, during an in situ bioprinting process. Based on the previous work carried out on the IMAGObot platform (a 5 Degrees of Freedom robotic manipulator), the aim is to print on moving and non-planar surfaces, following the trajectory of a fiducial marker placed onto the patient, inside the robot workspace. For this purpose, a monocular vision system (featured by a webcam and fiducial markers positioned in the robot environment) and a software interface communicating with the robot controller were developed. The control algorithm was entirely developed in the Python environment using the OpenCV library for marker pose estimation and used to update the robot trajectory concerning the detected marker motion on LinuxCNC software. Moreover, in order to mimic the physiological displacement of a patient's rib cage due to breathing, a moving 3D-printed platform and a silicone chest phantom were fabricated. The motion compensation system was tested by regenerating a defect on the chest phantom during the respiratory phase through extrusion based in situ bioprinting.

Target Recognition and Navigation Path Optimization Based on NAO Robot

The NAO robot integrates sensors, vision systems, and control systems. Its monocular vision system is adopted to locate the target object in the three-dimensional space of robots. Firstly, a positioning model based on monocular vision is established according to the principle of small hole perspective. Then, the position coordinates of the target center are obtained in the image coordinate system. In the model mentioned above, the relationship between position coordinates and image coordinates is established at a certain space height. According to this relationship, the two-dimensional coordinates in the image are converted into the three-dimensional coordinates in the robot coordinate system. After getting the target location, we establish the navigation map and find the optimal path under the unknown environment. Based on the simultaneous localization and the mapping (SLAM) theory, the sonar sensor of the NAO robot is used to detect the distance between the robot and the obstacles or between the robot and the end landmark. Moreover, the sonar sensor and the camera are used to distinguish the obstacle and the landmark. After the navigation map is built, the bi-directional parallel search strategy and the simulated annealing algorithm are introduced to improve the traditional artificial bee colony algorithm, and the improved artificial bee colony algorithm is proposed to find an optimal path in the navigation map. Finally, the experimental results show that based on the built environment map, the robot can find an optimal path from the origin to the landmark on the premise of avoiding obstacles.

The Design and the Development of a Biped Robot Cooperation System

The aim of this paper is to design a fuzzy motion control algorithm for a developed monocular vision system based on a cooperative transportation system of two humanoid robots. The control strategies of the cooperation transportation system contain three stages, including object searching, walking toward the transported object, and cooperatively moving the transported object. To have different moving speeds, the gait step size was pre-planned as two different modes, i.e., one of the gaits is selected to let the HR have large variations of motion and another gait is to make the HR with small variations. The fuzzy motion control algorithm is utilized to select the appropriate mode of gait. Both humanoid robots can actively search and move to the front of the target object, then cooperatively lift the target and carry it to the platform. The task of synchronous movement is controlled with fuzzy techniques through the control terminal. From the experimental results, it can be seen that both robots can distinguish the orientation of the target, move to the appropriate position, and then successfully raise the target together.