Three-dimensional World Coordinates Research Articles

Real-Time Recognition and Localization of Apples for Robotic Picking Based on Structural Light and Deep Learning

The apple is a delicious fruit with high nutritional value that is widely grown around the world. Apples are traditionally picked by hand, which is very inefficient. The development of advanced fruit-picking robots has great potential to replace manual labor. A major prerequisite for a robot to successfully pick fruits the accurate identification and positioning of the target fruit. The active laser vision systems based on structured algorithms can achieve higher recognition rates by quickly capturing the three-dimensional information of objects. This study proposes to combine the laser active vision system with the YOLOv5 neural network model to recognize and locate apples on trees. The method obtained accurate two-dimensional pixel coordinates, which, when combined with the active laser vision system, can be converted into three-dimensional world coordinates for apple recognition and positioning. On this basis, we built a picking robot platform equipped with this visual recognition system, and carried out a robot picking experiment. The experimental findings showcase the efficacy of the neural network recognition algorithm proposed in this study, which achieves a precision rate of 94%, an average precision mAP% of 92.86%, and a spatial localization accuracy of approximately 4 mm for the visual system. The implementation of this control method in simulated harvesting operations shows the promise of more precise and successful fruit positioning. In summary, the integration of the YOLOv5 neural network model with an active laser vision system presents a novel and effective approach for the accurate identification and positioning of apples. The achieved precision and spatial accuracy indicate the potential for enhanced fruit-harvesting operations, marking a significant step towards the automation of fruit-picking processes.

Construction of features database using different platform navigation solutions for aiding PDR algorithm

Nowadays, navigation can be accomplished based on several technologies. Some of these technologies rely solely on camera vision-based algorithms, whereas others rely solely on motion sensors’ data. Furthermore, some solutions combine vision-based algorithms with data from motion sensors. In this study, three platform navigation algorithms (stereo visual odometry technique, combining the distance calculated from the visual optical flow with estimated heading from gyroscope sensor measurements, and combining the distance calculated from the visual optical flow with estimated heading from magnetometer sensor measurements) were compared to select the most accurate among them to be employed for features database construction. This features database, which contains various well-described features with known three-dimensional world coordinates, can be utilized to help any indoor pedestrian dead reckoning (PDR) navigation solution as well as mapping purposes. Tests' results demonstrated that vision-based technology (stereo visual odometry technique) is more accurate than the others however it should be aided with additional constraints to improve navigation positional accuracy. Afterward, this chosen technology was used to construct a well-described features database which was checked to assess its accuracy.

Deep Learning-Based Violin Bowing Action Recognition

We propose a violin bowing action recognition system that can accurately recognize distinct bowing actions in classical violin performance. This system can recognize bowing actions by analyzing signals from a depth camera and from inertial sensors that are worn by a violinist. The contribution of this study is threefold: (1) a dataset comprising violin bowing actions was constructed from data captured by a depth camera and multiple inertial sensors; (2) data augmentation was achieved for depth-frame data through rotation in three-dimensional world coordinates and for inertial sensing data through yaw, pitch, and roll angle transformations; and, (3) bowing action classifiers were trained using different modalities, to compensate for the strengths and weaknesses of each modality, based on deep learning methods with a decision-level fusion process. In experiments, large external motions and subtle local motions produced from violin bow manipulations were both accurately recognized by the proposed system (average accuracy > 80%).

A stereovision model applied in bio-micromanipulation system based on stereo light microscope.

A bio-micromanipulation system is designed for manipulating micro-objects with a length scale of tens or hundreds of microns based on stereo light microscope. The world coordinate reconstruction of points on the surface of micro-objects is an important goal for the micromanipulation. Traditional pinhole camera model is applied widely in macrocomputer vision. However, this model will output bad data with remarkable error if it is directly used to reconstruct three-dimensional world coordinates for stereo light microscope. Therefore, a novel and improved pinhole camera model applied in bio-micromanipulation system is proposed in this article. The new model is composed of binocular-pinhole model and error-correction model. The binocular-pinhole model is used to output the basic world coordinates. The error-correction model is used to correct the errors from the basic world coordinates and outputs the final high-precision world coordinates. The results show that the new model achieves a precision of 0.01mm in the X direction, 0.01mm in the Y direction, and 0.015mm in the Z direction within a maximum reconstruction distance of 4.1mm in the X direction, 2.9mm in the Y direction, and 2.25mm in the Z direction, and that traditional pinhole camera model achieves a lower and unsatisfactory precision of about 0.1mm.

High-accuracy optical extensometer based on coordinate transform in two-dimensional digital image correlation

In the measurement of plate specimens, traditional two-dimensional (2D) digital image correlation (DIC) is challenged by two aspects: (1) the slant optical axis (misalignment of the optical camera axis and the object surface) and (2) out-of-plane motions (including translations and rotations) of the specimens. There are measurement errors in the results measured by 2D DIC, especially when the out-of-plane motions are big enough. To solve this problem, a novel compensation method has been proposed to correct the unsatisfactory results. The proposed compensation method consists of three main parts: 1) a pre-calibration step is used to determine the intrinsic parameters and lens distortions; 2) a compensation panel (a rigid panel with several markers located at known positions) is mounted to the specimen to track the specimen’s motion so that the relative coordinate transformation between the compensation panel and the 2D DIC setup can be calculated using the coordinate transform algorithm; 3) three-dimensional world coordinates of measuring points on the specimen can be reconstructed via the coordinate transform algorithm and used to calculate deformations. Simulations have been carried out to validate the proposed compensation method. Results come out that when the extensometer length is 400 pixels, the strain accuracy reaches 10 με no matter out-of-plane translations (less than 1/200 of the object distance) nor out-of-plane rotations (rotation angle less than 5°) occur. The proposed compensation method leads to good results even when the out-of-plane translation reaches several percents of the object distance or the out-of-plane rotation angle reaches tens of degrees. The proposed compensation method has been applied in tensile experiments to obtain high-accuracy results as well.

A visualization method of four-dimensional polytopes by oval display of parallel hyperplane slices

A method to visualize polytopes in a four-dimensional euclidian space (x, y, z, w) is proposed. A polytope is sliced by multiple hyperplanes that are parallel to each other and separated by uniform intervals. Since the hyperplanes are perpendicular to the w-axis, the resulting multiple slices appear in the three-dimensional (x, y, z) space and they are shown by the standard computer graphics. The polytope is rotated extrinsically in the four-dimensional space by means of a simple input method based on keyboard typings. The multiple slices are placed on a parabola curve in the three-dimensional world coordinates. The slices in a view window form an oval appearance. Both the simple and the double rotations in the four-dimensional space are applied to the polytope. All slices synchronously change their shapes when a rotation is applied to the polytope. The compact display in the oval of many slices with the help of quick rotations facilitate a grasp of the four-dimensional configuration of the polytope.

Research and Realization of a New Algorithm for Stereo Matching Based on Binocular Vision

In the development of a three-dimensional measurement system, binocular stereo matching is the most important and difficult. In the basis of introducing selective principles of matching algorithm, a new stereo matching algorithm for binocular vision is put forward that is named noncoded difference measuring distance. The algorithm effectively grapples with the problem of searching for the coincidence relation of raster and can efficiently and accurately obtain three-dimensional world coordinates of the entities. Experiment results show that this 3D measuring machine can effectively measure the 3D solid profile of free surface. During the evaluation test for accuracy, scan a standard plane. Fit all 3D points in one plane, and then the flatness value of this plane is obtained. The flatness value of the standard plane has been ultimately measured as: ± 0.0462mm, this measuring accuracy can completely satisfy the requirements of rapid prototyping or CNC machining, it as well as achieves the stated accuracy (± 0.05mm).

基于单幅二维图像的摄像机标定方法研究<br>A Camera Calibration Method Study Based on Single Two-Dimensional Image

为提高场景摄像机的标定精度和标定效率，本文利用摄像机透镜成像原理，基于针孔成像模型，提出了一种基于单幅二维图像的摄像机标定方法。该方法利用最小二乘法直接计算世界坐标系与计算机图像坐标系之间的数学模型，有效解决了摄像机内外参数难以直接计算，且计算精度不高的问题。方法简单有效，实现摄像机快速标定的同时保证了标定精度和可靠性，并提高了标定效率。通常情况下，当空间存在6个已知点的三维世界坐标及其计算机图像坐标时，即可实现摄像机标定。增加空间已知点的数量，还可进一步提高标定精度。利用大量的标定图像数据进行验证，结果表明，该标定算法计算精度高，计算量小，提高了摄像机标定的可靠性和精确性，具有一定的理论意义和实用价值。 In order to improve the calibration accuracy and efficiency of the camera, a camera calibration method based on single two-dimensional image is proposed in this paper. Method of least squares is used to calculate the mathematical model between the world coordinate system and the computer image coordinate system, so that the problem that the Internal and external camera parameters and difficult to calculate directly and that the calculation accuracy is not high can be solved effectively. Calibration accuracy and reliability are guaranteed and also the calibration efficiency is improved in this method. Under normal circumstances, when there are three-dimensional world coordinates and its computer image coordinates of six known points in the space, then camera calibration can be realized. The calibration accuracy can be further improved if the number of space known point is increased. A large number of calibration image data is tested to improve that this calibration method has high accuracy and small amount of calculation, and improves the reliability and accuracy of the camera calibration. This method has theoretical significance and practical value.

A performance evaluation of vision and radio frequency tracking methods for interacting workforce

Analysis and understanding of worksite operations requires the collection of trajectory information regarding active onsite entities. The trajectories provide essential information for assessment of site operations and management strategies. Unfortunately, many such tracking and assessment methods are based on manually generated trajectories, thereby excluding long term and complex scenarios. Automated trajectory extraction is a potential solution, however the automated method must provide reliable trajectory information. This paper evaluates the capability of two sensor technologies to provide accurate trajectory data. The primary system investigated is a vision-based strategy while the secondary system investigated is an Ultra Wideband System (UWB). A robotic total station (RTS) provides the ground truth of a single worker for evaluation of both sensor technologies. For comparison, the visual tracking signal is mapped to three-dimensional world coordinates. The UWB and RTS signals are filtered to remove outliers, and are synchronized with the visual tracking signal. Given the single target tracked by the RTS, both UWB and vision are shown to be feasible when the expected accuracy is 1 m. A second investigation seeks to demonstrate the accuracy of the proposed visual tracking system for tracking multiple interacting workforce, which relies on machine learning to generate target descriptions. The UWB signals for the personnel provide ground truth to evaluate the reliability and accuracy of the vision-based tracking algorithm. The proposed visual tracking system is reliable and accurate enough to permit automated extraction of trajectory information for analysis purposes, such as would be required for work sampling and productivity assessments.

Accurate measurement of three-dimensional deformations in deformable and rigid bodies using computer vision

Recently, digital-image-correlation techniques have been used to accurately determine two-dimensional in-plane displacements and strains. An extension of the two-dimensional method to the acquisition of accurate, three-dimensional surfacedisplacement data from a stereo pair of CCD cameras is presented in this paper. A pin-hole camera model is used to express the transformation relating three-dimensional world coordinates to two-dimensional computer-image coordinates by the use of camera extrinsic and intrinsic parameters. Accurate camera model parameters are obtained for each camera independently by (a) using several points which have three-dimensional world coordinates that are accurate within 0.001 mm and (b) using two-dimensional image-correlation methods that are accurate to within 0.05 pixels to obtain the computer-image coordinates of various object positions. A nonlinear, least-squares method is used to select the optimal camera parameters such that the deviations between the measured and estimated image positions are minimized. Using multiple orientations of the cameras, the accuracy of the methodology is tested by performing translation tests. Using theoretical error estimates, error analyses are presented. To verify the methodology for actual tests both the displacement field for a cantilever beam and also the surface, three-dimensional displacement and strain fields for a 304L stainless-steel compact-tension specimen were experimentally obtained using stereo vision. Results indicate that the three-dimensional measurement methodology, when combined with two-dimensional digital correlation for subpixel accuracy, is a viable tool for the accurate measurement of surface displacements and strains.