Camera Coordinate System Research Articles

Visual-Inertial Fusion-Based Five-Degree-of-Freedom Motion Measurement System for Vessel-Mounted Cranes

Vessel-mounted cranes operate in complex marine environments, where precise measurement of cargo positions and attitudes is a key technological challenge to ensure operational stability and safety. This study introduces an integrated measurement system that combines vision and inertial sensing technologies, utilizing a stereo camera and two inertial measurement units (IMUs) to capture cargo motion in five degrees of freedom (DOF). By merging data from the stereo camera and IMUs, the system accurately determines the cargo’s position and attitude relative to the camera. The specific methodology is introduced as follows: First, the YOLO model is adopted to identify targets in the image and generate bounding boxes. Then, using the principle of binocular disparity, the depth within the bounding box is calculated to determine the target’s three-dimensional position in the camera coordinate system. Simultaneously, the IMU measures the attitude of the cargo, and a Kalman filter is applied to fuse the data from the two sensors. Experimental results indicate that the system’s measurement errors in the x, y, and z directions are less than 2.58%, 3.35%, and 3.37%, respectively, while errors in the roll and pitch directions are 3.87% and 5.02%. These results demonstrate that the designed measurement system effectively provides the necessary motion information in 5-DOF for vessel-mounted crane control, offering new approaches for pose detection of marine cranes and cargoes.

A Calculation Method of Bearing Balls Rotational Vectors Based on Binocular Vision Three-Dimensional Coordinates Measurement

The rotational speed vectors of the bearing balls affect their service life and running performance. Observing the actual rotational speed of the ball is a prerequisite for revealing its true motion law and conducting sliding behavior simulation analysis. To address the need for accuracy and real-time measurement of spin angular velocity, which is also under high-frequency and high-speed ball motion conditions, a new measurement method of ball rotation vectors based on a binocular vision system is proposed. Firstly, marker points are laid on the balls, and their three-dimensional (3D) coordinates in the camera coordinate system are calculated in real time using the triangulation principle. Secondly, based on the 3D coordinates before and after the movement of the marker point and the trajectory of the ball, the mathematical model of the spin motion of the ball was established. Finally, based on the ball spin motion model, the three-dimensional vision measurement technology was first applied to the measurement of the bearing ball rotation vector through formula derivation, achieving the analysis of bearing ball rolling and sliding characteristics. Experimental results demonstrate that the visual measurement system with the frame rate of 100 FPS (frames per second) yields a measurement error within ±0.2% over a speed range from 5 to 50 RPM (revolutions per minute), and the maximum measurement errors of spin angular velocity and linear velocity are 0.25°/s and 0.028 mm/s, respectively. The experimental results show that this method has good accuracy and stability in measuring the rotation vector of the ball, providing a reference for bearing balls' rotational speed monitoring and the analysis of the sliding behavior of bearing balls.

Dynamic Scene Representation for Docking in Urban Waters Using a Stereo Camera

Abstract To navigate and dock safely in urban waters, an autonomous ferry must assess its safe navigation paths by identifying navigable free space and potential obstacles. A deep understanding of its surroundings is crucial for situational awareness. Knowing which objects are static and dynamic is an indispensable ingredient for path planning algorithms and collision avoidance systems, and is especially useful for docking operations, as moving obstacles can interfere with the optimal path to the dock. To handle all types of obstacles, we present a novel dynamic scene representation for docking in urban waters using a short baseline stereo camera. Obstacles protruding from the water surface are represented with vertically oriented rectangles, known as stixels. The stixels are tracked over consecutive frames with dense optical flow. Together with stereo correspondences, their 3D motion is estimated. Stixels representing the same object are grouped together into line segments which are classified as either dynamic or static based on the motion of its associated stixels. Our dynamic representation presents objects in the 3D camera coordinate system as a bird’s eye view map. Additionally, stereo depth uncertainty can significantly impact depth estimates and is considered using a Kalman filter, providing smooth position and velocity estimates. We test our dynamic scene representation on a real docking scenario recorded with an autonomous ferry using a stereo camera. We evaluate the accuracy of the reconstructed 3D scene points and the velocity estimates of the dynamic objects using LiDAR and GNSS.

MRCNN: Multi-input residual convolution neural network for three-dimensional reconstruction of bubble flows from light field images

Accurate measurement of bubbles in air-water two-phase flows holds immense significance in the realm of thermal hydraulics assessments within nuclear reactors. Nevertheless, conventional bubble measurement techniques grapple with challenges encompassing system intricacy, limited real-time capabilities, and inaccuracies stemming from their inherent two-dimensional (2-D) nature. In response, we pioneered an innovative three-dimensional (3-D) analysis approach that leverages light field (LF) imaging diagnosis and deep learning algorithms. Unlike traditional 2-D reconstruction methods, our approach enables direct computation of bubble depth from LF images using digital refocusing technology. Following calibration, a seamless transformation is established between the camera coordinate system and the real-world coordinate system using a sharpness evaluation algorithm. This calibration process ensures precise alignment of refocused images with real-world positions. Subsequently, fully automated and highly accurate computations of bubble depth are realized from input images via the incorporation of a multi-input residual convolution neural network (MRCNN). The limitations of traditional two-dimensional imaging techniques are effectively addressed by this methodology, resulting in a reduction in measurement errors. The study confirms the feasibility of employing LF imaging diagnosis and deep learning algorithms for bubble measurements in an air-water two-phase flow, offering a significant improvement over traditional methods.

Research on visual servo strategy for the uncalibrated non-standard manipulator

Abstract Aiming at the problem that the hand-eye calibration of the uncalibrated non-standard manipulator cannot be addressed by the general calibration method, i.e., the eye-in-hand and eye-to-hand calibration method of the visual servo system was proposed to guide the manipulator without calibration. First, a camera was installed on the manipulator to complete the eye-in-hand calibration and achieve transformation from the end coordinate system to the camera coordinate system. Second, a second camera was used to achieve eye-to-hand calibration and obtain the transformation relationship from the end coordinate system to the calibration board. The entire calibration process only requires the collection of images from two scenarios, i.e., the docking and the completion of the docking processes. The implementation results show that the rotation error between groups is less than 1.5°, and the translation error between groups is less than 8 mm. Therefore, the proposed visual servo strategy can accurately guide the uncalibrated non-standard manipulator.

Enhanced Real-Time Target Detection for Picking Robots Using Lightweight CenterNet in Complex Orchard Environments

The rapid development of artificial intelligence and remote sensing technologies is indispensable for modern agriculture. In orchard environments, challenges such as varying light conditions and shading complicate the tasks of intelligent picking robots. To enhance the recognition accuracy and efficiency of apple-picking robots, this study aimed to achieve high detection accuracy in complex orchard environments while reducing model computation and time consumption. This study utilized the CenterNet neural network as the detection framework, introducing gray-centered RGB color space vertical decomposition maps and employing grouped convolutions and depth-separable convolutions to design a lightweight feature extraction network, Light-Weight Net, comprising eight bottleneck structures. Based on the recognition results, the 3D coordinates of the picking point were determined within the camera coordinate system by using the transformation relationship between the image’s physical coordinate system and the camera coordinate system, along with depth map distance information of the depth map. Experimental results obtained using a testbed with an orchard-picking robot indicated that the proposed model achieved an average precision (AP) of 96.80% on the test set, with real-time performance of 18.91 frames per second (FPS) and a model size of only 17.56 MB. In addition, the root-mean-square error of positioning accuracy in the orchard test was 4.405 mm, satisfying the high-precision positioning requirements of the picking robot vision system in complex orchard environments.

Joint stereo 3D object detection and implicit surface reconstruction

We present a new learning-based framework S-3D-RCNN that can recover accurate object orientation in SO(3) and simultaneously predict implicit rigid shapes from stereo RGB images. For orientation estimation, in contrast to previous studies that map local appearance to observation angles, we propose a progressive approach by extracting meaningful Intermediate Geometrical Representations (IGRs). This approach features a deep model that transforms perceived intensities from one or two views to object part coordinates to achieve direct egocentric object orientation estimation in the camera coordinate system. To further achieve finer description inside 3D bounding boxes, we investigate the implicit shape estimation problem from stereo images. We model visible object surfaces by designing a point-based representation, augmenting IGRs to explicitly address the unseen surface hallucination problem. Extensive experiments validate the effectiveness of the proposed IGRs, and S-3D-RCNN achieves superior 3D scene understanding performance. We also designed new metrics on the KITTI benchmark for our evaluation of implicit shape estimation.

Research on a visual positioning method of paddy field weeding wheels based on laser rangefinder-camera calibration

The positioning of paddy field weeding wheels is of great significance for compensating the operation deviation between the weeding wheel and the seedling row in paddy field mechanical weeding. In this paper, a visual positioning method for determining the three-dimensional coordinates of weeding wheels in the camera coordinate system is proposed. By fixing the laser rangefinder with the weeding wheel, the proposed method converts the positioning of weeding wheels into solving the relative pose relationship between the laser rangefinder and the camera. Then, based on the constraints of the laser spots on the AprilTag calibration plane, a nonlinear optimization model is established to obtain the relative pose parameters. In the experiment, two evaluation indicators were proposed to evaluate the calibration accuracy. The experimental results showed that the proposed visual positioning method of the weeding wheel can reach a mean positioning error of 2.766 mm and a mean pixel error of 7.161 pixels.

Research and Implementation of Robotic Arm Positioning and Grasping Based on Visual Guidance

Abstract This paper proposes a robotic arm vision guidance system based on the ROS platform, which combines the Hikrobot Robotics 2D vision sensor with the Xinsong GCR5 robotic arm to successfully realise vision-guided robotic arm positioning and grasping. Camera calibration, hand-eye calibration, and deep learning techniques are used to establish an accurate transformation relationship between the base coordinate system of the robotic arm and the camera coordinate system to achieve efficient target goal recognition and localisation. Adopting the MoveIt framework and RRT-connect algorithm, the robotic arm can flexibly plan the movement path and successfully complete the target grasping, which provides the theoretical basis and practical experience for robotic arm intelligent grasping.

TQU-SLAM Benchmark Dataset for Comparative Study to Build Visual Odometry Based on Extracted Features from Feature Descriptors and Deep Learning

The problem of data enrichment to train visual SLAM and VO construction models using deep learning (DL) is an urgent problem today in computer vision. DL requires a large amount of data to train a model, and more data with many different contextual and conditional conditions will create a more accurate visual SLAM and VO construction model. In this paper, we introduce the TQU-SLAM benchmark dataset, which includes 160,631 RGB-D frame pairs. It was collected from the corridors of three interconnected buildings comprising a length of about 230 m. The ground-truth data of the TQU-SLAM benchmark dataset were prepared manually, including 6-DOF camera poses, 3D point cloud data, intrinsic parameters, and the transformation matrix between the camera coordinate system and the real world. We also tested the TQU-SLAM benchmark dataset using the PySLAM framework with traditional features such as SHI_TOMASI, SIFT, SURF, ORB, ORB2, AKAZE, KAZE, and BRISK and features extracted from DL such as VGG, DPVO, and TartanVO. The camera pose estimation results are evaluated, and we show that the ORB2 features have the best results (Errd = 5.74 mm), while the ratio of the number of frames with detected keypoints of the SHI_TOMASI feature is the best (rd=98.97%). At the same time, we also present and analyze the challenges of the TQU-SLAM benchmark dataset for building visual SLAM and VO systems.

Hand-eye calibration of line structured-light sensor based on double-sphere constraints

Aiming at the complex problem of solving the positional relationship between the line laser sensor at the end of the robot wrist and the coordinate system of the end flange, a line laser hand-eye calibration method based on a double ball fixed target is proposed. Firstly, a double standard ball with known radius and center distance is used as a reference tool, and the point cloud data of a single contour line is collected several times, and the coordinates of the two ball centers under the camera coordinate system can be obtained according to the center coordinates of the circle obtained from the fitted circle combined with the Pythagoras theorem. Then, according to the principle that the two spherical centers are fixed under the robot base coordinate system, the rigid transformation matrix between the line laser sensor and the robot is successively solved. When solving the rotation matrix, the constraint of the distance between the spherical centers of the two balls is introduced to obtain a solution with better accuracy. Experimental results show that the method has better overall performance compared with the traditional single-ball hand-eye calibration method.

An omnidirectional spatial monocular visual localization and tracking method for indoor unmanned aerial vehicles based on the two-axis rotary table

Aiming at the complexity and poor adaptability of the calibration process in the traditional unmanned aerial vehicles (UAV) indoor visual positioning, this paper proposes an omnidirectional spatial tracking and localization method for indoor UAV based on the two-axis rotary table. Firstly, the position of the UAV fuselage feature points in the camera coordinate system of the turntable camera is computed by the Pespective-n-Point algorithm utilizing known position information of a plurality of feature points with pixel coordinate information in the corresponding image. Pixel coordinate information is extracted by obtaining UAV body-specific feature points from rotary table camera shots. Then, UAV localization in omnidirectional space can be obtained by using the calibrated rotary axis parameters of the rotary table and the rotation angle of the rotary table and substituting them into Rodriguez’s formula to unify the UAV position information acquired by the rotary table camera at different positions into a unified coordinate system. Finally, the angle at which the rotary table should rotate is calculated from the obtained UAV pose and the spatial position of the camera optical center and the rotary axis of the rotary table. The calculated angle is fed back to the turntable as feedback information. The rotary table receiving the feedback information is rotated to a position where the UAV is located at the center of the camera image. Thereby the tracking and localization of the UAV is realized. The experimental results show that the spatial range of localization is greatly expanded with the localization accuracy reaching the level of binocular visual localization. The omnidirectional spatial tracking and localization of indoor UAV can be conveniently realized by this method.

An identification algorithm of lateral correction amount for the weeding components in paddy fields based on multi-sensor fusion

In this paper, an identification algorithm of lateral correction amount for the weeding components in paddy fields based on multi-sensor fusion is proposed, which can accurately obtain the lateral deviation between the weeding components and the seedling rows under different soil hardness in paddy fields to avoid crushing seedlings. The proposed method first fuses the RGB images with depth images to obtain the three-dimensional point cloud of seedlings, establishes a visual calibration system to calibrate the positions of the weeding component at the limit positions in the camera coordinate system, then obtains the relative pose relationship between the camera coordinate system and the ground coordinate system based on the inertial measurement unit to solve the influence of the altitude change of the camera on the identification of the lateral correction amount, and finally calculates the lateral correction amount based on the lateral deviation model in the ground coordinate system. The experimental platforms for the visual calibration of the weeding components and the identification of the lateral correction amount were established. The experimental results showed that the mean positioning error of the weeding components was 2.766 mm, the mean identification error of the lateral correction amount did not exceed 0.22 cm, and the standard deviation of the identification error did not exceed 0.18 cm.

Experimental and simulation investigation of stereo-DIC via a deep learning algorithm based on initial speckle positioning technology.

For the deep-learning-based stereo-digital image correlation technique, the initial speckle position is crucial as it influences the accuracy of the generated dataset and deformation fields. To ensure measurement accuracy, an optimized extrinsic parameter estimation algorithm is proposed in this study to determine the rotation and translation matrix of the plane in which the speckle is located between the world coordinate system and the left camera coordinate system. First, the accuracy of different extrinsic parameter estimation algorithms was studied by simulations. Subsequently, the dataset of stereo speckle images was generated using the optimized extrinsic parameters. Finally, the improved dual-branch CNN deconvolution architecture was proposed to output displacements and strains simultaneously. Simulation results indicate that DAS-Net exhibits enhanced expressive capabilities, as evidenced by a reduction in displacement errors compared to previous research. The experimental results reveal that the mean absolute percentage error between the stereo-DIC results and the generated dataset is less than 2%, suggesting that the initial speckle positioning technology effectively minimizes the discrepancy between the images in the dataset and those obtained experimentally. Furthermore, the DAS-Net algorithm accurately measures the displacement and strain fields as well as their morphological characteristics.

Spatial quadric calibration method for multi-line laser based on diffractive optical element

The traditional multi-line laser calibration method relies primarily on calibrating the spatial plane equations of multiple laser lines and reconstructing the multi-line laser in three dimensions through the plane equation and camera parameters. In traditional methods, the multi-line laser is projected as a straight line by default, but in reality, the laser line projected by the multi-line laser is not a plane due to the Diffractive Optical Element (DOE). Instead, it is a surface with a certain curvature in space. During the production process, the multi-line laser experiences significant distortion because of the DOE projecting at a large angle. Consequently, if traditional methods are used, accurate calibration of the multi-line laser cannot be achieved. To address this issue, this paper introduces a new calibration algorithm for multi-line laser based on a spatial quadric equation. By calibrating the spatial quadric equation of the multi-line laser, the spatial characteristics of the laser can be accurately captured. The algorithm involves projecting a crossed 7-line laser onto a calibration plate and changing the position of the calibration plate while capturing images. This allows for obtaining the spatial three-dimensional coordinates of each laser line in the camera coordinate system. Next, spatial quadric equations are fitted for each laser line based on the three-dimensional coordinates. Finally, 3D reconstruction is performed using the quadratic surface parameters of the multi-line laser and camera parameters. The results from 3D reconstruction demonstrate that, compared to traditional laser calibration methods based on planes, the proposed algorithm achieves higher calibration accuracy. It also improves the number of successful binocular multi-line laser matches and overall accuracy.

Automatic calibration and association for roadside radar and camera based on fluctuating traffic volume

Accurate perception of the movement and appearance of vehicles depends on the robustness and reliability of the extrinsic parameters calibration in a multi-sensor fusion scenario. However, conventional calibration methods require manual acquisition of prior information, leading to high labor costs and low calibration accuracy. Therefore, we proposed an automatic coarse-to-fine calibration method for roadside radar and camera sensors to lower costs and improve accuracy. Next, an association strategy based on fluctuating traffic volumes was also developed to assist in robust target matching during the coarse-to-fine calibration process. Finally, extrinsic parameters between the radar coordinate system and camera coordinate system were calibrated through double rotations of the position vectors obtained from each system. To verify the proposed method, an experiment was conducted on a pedestrian bridge using an uncalibrated 4D millimeter-wave radar and a traffic monocular camera. The results showed that our proposed method reduced the interquartile range of the roll angle by 41.5% compared to a state-of-the-art neural network method. It also outperformed the manual calibration method by 2.47% in terms of the average reprojection error.

Vision‐based fatigue crack automatic perception and geometric updating of finite element model for welded joint in steel structures

AbstractDigital twin requires establishing a self‐updated model to simulate the structural damage perceived onsite. Despite the great success in damage identification and quantification, the difficulty in registration still limits the efficiency of model updating. This study presented a framework that enables a finite element (FE) model of welded joints to remesh itself for updating the geometric changes caused by the fatigue crack. Leveraging the linear geometry of the weld, a crack registration algorithm was proposed for the automation of crack perception. First, a dual‐task network was established to identify the crack and weld on the 2D image, where the deep Hough transform was introduced to detect the positioning weld among the irregular structural geometry. With the time‐of‐flight technique, the crack was then reconstructed and quantified in the 3D camera coordinate system. Meanwhile, the 3D structure coordinate system was established from positioning welds. Through simple coordinate transformation, the fatigue crack was automatically registered to the welded joint. Finally, the perception algorithms were integrated with the FE model, taking about 1 min to map the crack into the model. Under laboratory tests, the perception performance was not sensitive to the camera pose. The perceived errors were mainly reflected in the crack local morphology, not leading to improper reconstruction of the structural stiffness matrix.

Grasp and Inspection of Mechanical Parts based on Visual Image Recognition Technology

In the process of industrial production, whether the machine can work stably is an important issue related to production efficiency and production safety. The cleaning of mechanical parts is an important factor affecting the stable operation of mechanical equipment, so the cleaning of mechanical parts is one of the important manufacturing links before assembly and manufacturing. The important value of the application of image recognition technology in the quality inspection of mechanical parts is briefly analyzed, and then the common problems in the quality inspection of mechanical parts are studied, the relevant measures of the application of image recognition technology in the quality inspection of mechanical parts are discussed, and finally the case analysis of the quality inspection of mechanical parts is carried out. The main purpose is to rationally use image recognition technology to complete the quality inspection of mechanical parts, so as to improve the overall level of production automation and give full play to the maximum role of this technology. At present, a variety of mechanical parts classification methods based on deep learning are studied. According to the demand of industrial machine parts, the corresponding machine parts data set is created. According to the production requirements, the SPP structure in the traditional YOLOV5 network is improved, and a new SPC structure is proposed: The YOLOv5 network with SPC structure is characterized by no pooling layer, which improves the accuracy and recall rate compared with the traditional YOLOv5 network. Secondly, the control method of robot grasping mechanical parts is studied. Using the camera to capture the image and the upper computer to calculate the mechanical parts under the camera coordinate system; Through camera calibration and hand-eye calibration, the transformation relationship between the grasping coordinates of mechanical parts in the camera coordinate system and the grasping coordinates of mechanical parts in the robot base coordinate system is obtained. The grasping coordinates and part type information of the mechanical parts in the robot base coordinate system are transmitted from the host computer to the robot through Modbus/TCP communication protocol.

A multi-sensor fusion framework with tight coupling for precise positioning and optimization

In the dynamic landscape of artificial intelligence and robotics, the pursuit of accurate positioning in mobile robots has intensified. This research addresses the limitations of single-sensor SLAM (Simultaneous Localization and Mapping) techniques in complex settings by harnessing the collective strengths of LiDAR (Light Detection And Ranging), Camera, IMU (Inertial Measurement Unit), and GNSS (Global Navigation Satellite System) sensors. The proposed multi-sensor tightly-coupled SLAM framework is an integration of point-line feature-based laser–visual–inertial odometry, visual–laser fusion loop closure detection, and factor graph-based back-end optimization. Within the visual–inertial subsystem, an advanced LSD (Line Segment Detector) feature extraction strategy is introduced, incorporating point-line fusion to enhance visual line features. Additionally, the laser point cloud is projected onto the camera coordinate system, establishing depth associations with visual attributes. Strengthening the robustness of the visual–inertial subsystem in low-texture environments, camera poses undergo optimization through a sliding-window bundle adjustment method. In the laser-inertial subsystem, IMU preintegration mitigates laser point cloud distortion. Extracting edge and plane features, coupled with frame-to-local-map matching, enhances matching efficiency while streamlining computational intricacies. This amalgamation forms the basis of the laser–visual–inertial odometry fusion system. To overcome the limitations of standalone visual and laser-based loop closure detection, a dual-loop closure method utilizing visual–laser fusion is proposed. Leveraging the DBoW2 bag-of-words model, complemented by temporal–spatial consistency checks, enhances detection efficiency and accuracy. The integration of GNSS factors imparts global constraints for expansive outdoor scenarios. Employing factor graph-based back-end optimization, the refinement of laser–visual–inertial odometry factors, visual–inertial odometry factors, IMU preintegration factors, loop closure factors, and GNSS factors culminates in precise global pose estimation and high-fidelity point cloud maps. Through rigorous evaluation of the M2DGR dataset and a mobile robot platform, the proposed methodology emerges as an exemplar of performance, showcasing superiority over the state-of-the-art LIO-SAM technique. Achieving a reduction of 2.86m and 3.23m in the root mean square error of absolute pose estimation across divergent environments, this approach exhibits remarkable efficacy in outdoor scenarios, thereby elevating the precision and resilience of SLAM algorithms for mobile robots.

A three-dimensional vision measurement method based on double-line combined structured light

In this paper, a structured light vision measurement method using a scanning laser line and a positioning laser line is proposed. The novel method enables the scanning laser plane to slide along a slide rail while maintaining intersection with the positioning laser plane, eliminating the need to determine the scanning direction and moving step. During the measurement process, the laser plane equations need to be recalibrated for each new position, so a real-time calibration method is given. Initially, the geometric barycenter method is employed to detect the subpixel coordinates of the light stripe intersection point. Subsequently, these coordinates are projected into the camera coordinate system using the initial equations of the positioning laser plane. Finally, leveraging the normal information of the initial equation of the scanning laser plane and the three-dimensional coordinates of the light stripe intersection point, the real-time calibration of the scanning laser plane equations can be accomplished. The proposed method enables the three-dimensional reconstruction of objects, and its accuracy is verified through measurements on gauge blocks. Experimental results demonstrate that this method achieves precise and stable three-dimensional reconstruction of object surface shape.