Binocular Camera Research Articles

Extraction of Corn Plant Phenotypic Parameters with Keypoint Detection and Stereo Images

Corn is a global crop that requires the breeding of superior varieties. A crucial aspect of the breeding process is the accurate extraction of phenotypic parameters from corn plants. The existing challenges in phenotypic parameter extraction include low precision, excessive manual involvement, prolonged processing time, and equipment complexity. This study addresses these challenges by opting for binocular cameras as the data acquisition equipment. The proposed stereo corn phenotype extraction algorithm (SCPE) leverages binocular images for phenotypic parameter extraction. The SCPE consists of two modules: the YOLOv7-SlimPose model and the phenotypic parameter extraction module. The YOLOv7-SlimPose model was developed by optimizing the neck component, refining the loss function, and pruning the model based on YOLOv7-Pose. This model can better detect bounding boxes and keypoints with fewer parameters. The phenotypic parameter extraction module can construct the skeleton of the corn plant and extract phenotypic parameters based on the coordinates of the keypoints detected. The results showed the effectiveness of the approach, with the YOLOv7-SlimPose model achieving a keypoint mean average precision (mAP) of 96.8% with 65.1 million parameters and a speed of 0.09 s/item. The phenotypic parameter extraction module processed one corn plant in approximately 0.2 s, resulting in a total time cost of 0.38 s for the entire SCPE algorithm to construct the skeleton and extract the phenotypic parameters. The SCPE algorithm is economical and effective for extracting phenotypic parameters from corn plants, and the skeleton of corn plants can be constructed to evaluate the growth of corn as a reference. This proposal can also serve as a valuable reference for similar functions in other crops such as sorghum, rice, and wheat.

Height estimation of sugarcane tip cutting position based on multimodal alignment and depth image fusion

Sugarcane tip cutting is essential to reducing the rate of impurities in the harvest. To achieve adaptive regulation of the tip-cutting position by a sugarcane harvester, we propose a method for estimating the height of the tip-cutting position of sugarcane. The RGB and Binocular depth cameras are aligned to process the sugarcane tip region image. This involves threshold segmentation, morphological operations, and contour detection to identify the tip-cutting position and upper boundary contours. The depth image is segmented using contour pixel information and merged to form a colour depth image of the sugarcane's tip. This image is then transformed using depth data and triangular parallax principles to determine the height of the sugarcane tip-cutting position. The proposed method was evaluated in various sugarcane plantation environments. Comparative analysis between the proposed method and manual measurements of actual cutting position heights revealed that the RMSE ranged from 1.22 cm to 1.78 cm, and R2 varied between 0.79 and 0.86. These results demonstrate the effectiveness of the proposed method in accurately extracting the height information of sugarcane tip-cutting positions, which has a specific application value for the adaptive adjustment of the tip-cutting device of the sugarcane harvester.

Generation of Virtual Ground Control Points Using a Binocular Camera

This paper presents a methodology for generating virtual ground control points (VGCPs) using a binocular camera mounted on a drone. We compare the measurements of the binocular and monocular cameras between the classical method and the proposed one. This work aims to decrease human processing times while maintaining a reduced root mean square error (RMSE) for 3D reconstruction. Additionally, we propose utilizing COLMAP to enhance reconstruction accuracy by solely utilizing a sparse point cloud. The results demonstrate that implementing COLMAP for pre-processing reduces the RMSE by up to 16.9% in most cases. We prove that VGCPs further reduce the RMSE by up to 61.08%.

Development of a combined harvester navigation control system based on visual simultaneous localization and mapping-inertial guidance fusion

Recently, the existing unmanned systems of combine harvesters mostly adopts satellite navigation scheme, lacking real-time observation of harvesting adjustment. To improve the operational efficiency of combine harvester assisted navigation operation, this paper designs a combine harvester navigation control system based on vision simultaneous localization and mapping (SLAM)-inertial guidance fusion. The system acquires field image information and extracts the crop boundary line as the navigation datum by binocular camera. First, the system acquires field image information through binocular camera and extracts the crop boundary line as the navigation datum. Second, fusing camera and inertial guidance information to obtain the real-time relative position of a combine harvester. Third, constrained optimization of image and inertial guidance information is achieved through a sliding window optimization method based on tightly coupled nonlinear optimization. Finally, obtain the position of the combine harvester relative to the navigation datum line, and output a signal to the steering mechanism to realize the combine harvester in the field intelligent positioning navigation control. The system consists of binocular camera, inertial measurement unit, motorized steering wheel, monitor display, angle sensor and microcontroller. During field testing, the system underwent repetitive harvesting trials over a distance of 25 m.. The testing machine performs field operations at a speed of 0.9-1.5 m/s, with an average lateral deviation range of 2.21-8.62 cm, a standard deviation range of 0.13-4.21 cm and an average cutting rate range of 92.2%-96.0%, achieving the expected harvesting effect.

Freehand 3D Ultrasound Imaging Based on Probe-mounted Vision and IMU System

ObjectivesFreehand three-dimensional (3D) ultrasound (US) is of great significance for clinical diagnosis and treatment, it is often achieved with the aid of external devices (optical and/or electromagnetic, etc.) that monitor the location and orientation of the US probe. However, this external monitoring is often impacted by imaging environment such as optical occlusions and/or electromagnetic (EM) interference. MethodsTo address the above issues, we integrated a binocular camera and an inertial measurement unit (IMU) on a US probe. Subsequently, we built a tight coupling model utilizing the unscented Kalman algorithm based on Lie groups (UKF-LG), combining vision and inertial information to infer the probe's movement, through which the position and orientation of the US image frame are calculated. Finally, the volume data was reconstructed with the voxel-based hole-filling method. ResultsThe experiments including calibration experiments, tracking performance evaluation, phantom scans, and real scenarios scans have been conducted. The results show that the proposed system achieved the accumulated frame position error of 3.78 mm and the orientation error of 0.36° and reconstructed 3D US images with high quality in both phantom and real scenarios. ConclusionsThe proposed method has been demonstrated to enhance the robustness and effectiveness of freehand 3D US. Follow-up research will focus on improving the accuracy and stability of multi-sensor fusion to make the system more practical in clinical environments.

Multi‐object road waste detection and classification based on binocular vision

AbstractA road multi‐object detection algorithm is one of the core algorithms for intelligent road cleaning robots relying on machine vision. Most existing object detection algorithms analyse all image regions and finally calculate the category and location of each object. However, it is not necessary to analyse all areas of the image when detecting objects on the road surface where the background changes little, and the number of objects is small because there will be a lot of invalid calculations. If we can perform targeted local analysis on images instead of analysing all image regions, it will improve the detection efficiency. Therefore, this paper proposes a multi‐object detection method using a binocular camera and a convolutional neural network (CNN) that effectively reduces invalid calculations during the detection and improves detection efficiency. In the developed method, the binocular vision image acquired by the binocular camera is stereo matched and equalized, while linear regression and coordinate transformation eliminate the angle of the camera pair concerning the road surface. Then, the coordinates of the regions of interest (ROI) is calculated in the left vision image and the features within the ROI is extracted from the corresponding CNN's feature map. Next, ROI pooling resizes the extracted feature maps of different sizes to the same size, which are then input to the fully connected layers to output the results. The proposed binocular network and faster R‐CNN (VGG16) are trained and tested on a dataset involving 1000 road waste images. The experimental results demonstrate that the developed binocular network improves the detection accuracy and speed by 28.56% and 78.39%, respectively, compared with faster R‐CNN (VGG16), providing a reliable basis for a machine vision‐based intelligent road cleaning robot.

Study of Underwater Large-Target Localization Based on Binocular Camera and Laser Rangefinder

Currently, for underwater close-range large-target localization, visual localization techniques fail since large targets completely occupy the camera’s field of view at ultraclose ranges. To address the issue, a multi-stage optical localization method combining a binocular camera and a single-point laser rangefinder is proposed in this paper. The proposed method comprises three parts. First, the imaging model of the underwater camera is modified, and a laser rangefinder is used to further correct the underwater calibration results of the binocular camera. Second, YOLOv8 is applied to recognize the targets to prepare for target localization. Third, extrinsic calibration of the binocular camera and laser rangefinder is performed, and a Kalman filter is employed to fuse the target position information measured by the binocular camera and laser rangefinder. The experimental results show that, compared with using a binocular camera alone, the proposed method can accurately and stably locate the target at close ranges with an average error of only 2.27 cm, without the risk of localization failure, and reduces binocular localization error by 90.57%.

High-precision binocular camera calibration method based on a 3D calibration object.

A high-precision binocular camera calibration method is proposed to address the issues of poor calibration accuracy and large calibration errors in current practical applications. This method uses a triangular stereo sphere as the calibration object and employs steps, such as ellipse fitting, Cholesky decomposition, homography matrix solution, and nonlinear optimization, to compute the intrinsic and extrinsic parameters, distortion parameters, and relative pose of the binocular camera. Moreover, this method simplifies the correspondences between primitives, enabling simultaneous calibration of multiple viewpoint cameras. This method is also suitable for both binocular cameras consisting of two different structured monocular cameras and those composed of two image sensors within the same structure. Experimental results showed that this method outperforms traditional algorithms in terms of binocular camera calibration accuracy, calibration errors between left and right cameras, and robustness, resulting in a significant improvement in overall algorithm performance.

A Coupled Calibration Method for Dual Cameras-Projector System with Sub-Pixel Accuracy Feature Extraction.

Binocular structured light systems are widely used in 3D measurements. In the condition of complex and local highly reflective scenes, to obtain more 3D information, binocular systems are usually divided into two pairs of devices, each having a Single Camera and a Projector (SCP). In this case, the binocular system can be seen as Dual Cameras-Projector (DCP) system. In the DCP calibration, the Left-SCP and Right-SCP need to be calibrated separately, which leads to inconsistent parameters for the same projector, thus reducing the measurement accuracy. To solve this problem and improve manoeuvrability, a coupled calibration method using an orthogonal phase target is proposed. The 3D coordinates on a phase target are uniquely determined by the binocular camera in DCP, rather than being calculated separately in each SCP. This ensures the consistency of the projector parameters. The coordinates of the projector image plane are calculated through the unwrapped phase, while the parameters are calibrated by the plane calibration method. In order to extract sub-pixel accuracy feature points, a method based on polynomial fitting using an orthogonal phase target is exploited. The experimental results show that the reprojection error of our method is less than 0.033 pixels, which improves the calibration accuracy.

A non-contact method for weighting of aggregate pile based on binocular vision

Concrete is indispensable in contemporary industry, and the weighing of aggregates directly determines the quality of concrete preparation. In response to the difficulty of traditional methods in meeting the industrial requirements for accuracy, efficiency, and cost of measurement, this paper proposes a non-contact bone pile weight measurement method based on binocular vision. Firstly, the on-site image is obtained through a binocular camera, and the camera parameters are used to improve the image edge information and preprocess the image. Then, the improved semi global matching (SGM) algorithm is used to improve computational efficiency, obtaining three-dimensional information of the aggregate pile for volume calculation. Based on the watershed algorithm and T-S inference, the aggregate clearance rate is calculated, and the initial volume is corrected. Finally, a linear relationship is established between the density of the aggregate pile and the clearance rate, and the weight of the aggregate pile can be obtained by the density formula. The experimental results on different sizes of aggregate piles shows that the average error of the experiment is less than 4%. Efficiency of improved SGM matching algorithm has increased several times compared to the original algorithm, providing a low-cost and efficient measurement method for industrial aggregate weighing.

An Apple Detection and Localization Method for Automated Harvesting under Adverse Light Conditions

The use of automation technology in agriculture has become particularly important as global agriculture is challenged by labor shortages and efficiency gains. The automated process for harvesting apples, an important agricultural product, relies on efficient and accurate detection and localization technology to ensure the quality and quantity of production. Adverse lighting conditions can significantly reduce the accuracy of fruit detection and localization in automated apple harvesting. Based on deep-learning techniques, this study aims to develop an accurate fruit detection and localization method under adverse light conditions. This paper explores the LE-YOLO model for accurate and robust apple detection and localization. The traditional YOLOv5 network was enhanced by adding an image enhancement module and an attention mechanism. Additionally, the loss function was improved to enhance detection performance. Secondly, the enhanced network was integrated with a binocular camera to achieve precise apple localization even under adverse lighting conditions. This was accomplished by calculating the 3D coordinates of feature points using the binocular localization principle. Finally, detection and localization experiments were conducted on the established dataset of apples under adverse lighting conditions. The experimental results indicate that LE-YOLO achieves higher accuracy in detection and localization compared to other target detection models. This demonstrates that LE-YOLO is more competitive in apple detection and localization under adverse light conditions. Compared to traditional manual and general automated harvesting, our method enables automated work under various adverse light conditions, significantly improving harvesting efficiency, reducing labor costs, and providing a feasible solution for automation in the field of apple harvesting.

Binocular Visual Measurement Method Based on Feature Matching.

To address the issues of low measurement accuracy and unstable results when using binocular cameras to detect objects with sparse surface textures, weak surface textures, occluded surfaces, low-contrast surfaces, and surfaces with intense lighting variations, a three-dimensional measurement method based on an improved feature matching algorithm is proposed. Initially, features are extracted from the left and right images obtained by the binocular camera. The extracted feature points serve as seed points, and a one-dimensional search space is established accurately based on the disparity continuity and epipolar constraints. The optimal search range and seed point quantity are obtained using the particle swarm optimization algorithm. The zero-mean normalized cross-correlation coefficient is employed as a similarity measure function for region growing. Subsequently, the left and right images are matched based on the grayscale information of the feature regions, and seed point matching is performed within each matching region. Finally, the obtained matching pairs are used to calculate the three-dimensional information of the target object using the triangulation formula. The proposed algorithm significantly enhances matching accuracy while reducing algorithm complexity. Experimental results on the Middlebury dataset show an average relative error of 0.75% and an average measurement time of 0.82 s. The error matching rate of the proposed image matching algorithm is 2.02%, and the PSNR is 34 dB. The algorithm improves the measurement accuracy for objects with sparse or weak textures, demonstrating robustness against brightness variations and noise interference.

Extrinsic calibration for large FOV based on inverse depth parameterized bundle adjustment

To overcome the reliance on high-precision calibration plates in camera calibration, this paper proposes an extrinsic calibration method suitable for three-dimensional digital image correlation (3D-DIC) in large field of view (FOV). The method combines image feature algorithms with DIC techniques to extract matching point pairs (MPPs) from the left and right images of binocular cameras. These MPPs are then homogenized within the FOV. Next, initial values of the extrinsic parameters are solved based on epipolar constraint theory. Finally, the calibration parameters are nonlinearly optimized using the bundle adjustment method. To achieve stable and reliable numerical optimization in large FOV applications, the calibration control points (CCPs)’ spatial coordinates are represented using inverse depth parameterization. In scenarios where there might be a lack of sufficient CCPs, speckle patterns are artificially introduced to supplement the scene features. However, there is a lack of reliable experimental basis on how to add CCPs within the FOV. Therefore, through simulations, the factors affecting calibration accuracy are analyzed to guide the actual calibration process. The proposed method’s reliability and accuracy in large FOV 3D measurement are verified through experiments.

A multi‐degree‐of‐freedom monitoring method for slope displacement based on stereo vision

AbstractThree‐dimensional displacement monitoring over long distances has been a long‐standing concern in the structural health monitoring industry. In this study, a multi‐degree‐of‐freedom slope displacement monitoring method is developed by fusing computer vision and the 3D point triangulation method. Attributed to this method, the problems of outdoor binocular camera calibration, multi‐target mismatching, and outdoor illumination effects were solved. First, a two‐stage camera calibration method is proposed to accurately calibrate intrinsic and extrinsic camera parameters under a large field of view and long working distance conditions. Second, the adaptive spatial‐frequency method is proposed to calculate the coding and pixel coordinates of the monitored target. In this step, to solve the problem of mismatching monitored points in different camera frames, the Augmented Reality University of Cordoba code is introduced to provide a unique identity code for each monitored point. To mitigate the impact of illumination and other factors on pixel coordinate calculation, an adaptive pixel coordinate calculation method that combines information from the spatial and frequency domains is proposed., Third, based on the intrinsic and extrinsic parameters of the stereo camera and the pixel coordinates of the monitored points, the 3D coordinates of the monitored points are obtained through triangulation. Finally, the accuracy experiments and stability experiments are conducted. According to the results of the experiments, the measurement distance is positively correlated with the measurement error. And the baseline length is negatively correlated with the measurement error in the z‐direction. Ultimately, we suggest that the ratio of baseline length to measurement distance should be greater than 40%. When the recommended value is satisfied, the measurement error is less than 1 mm when the measurement distance is less than 40 m. When the measurement distance is equal to 90 m, the measurement error is less than 5 mm. Meanwhile, stability experiments of the algorithm were carried out, and in a period of outdoor validation experiments, the fluctuations were only sub‐millimeter, demonstrating good anti‐interference performance. Moreover, the method proposed in this study successfully monitored a landslide disaster in Guangxi, which demonstrated its outstanding practical application capabilities.

A Pavement Crack Detection Method via Deep Learning and a Binocular-Vision-Based Unmanned Aerial Vehicle

This study aims to enhance pavement crack detection methods by integrating unmanned aerial vehicles (UAVs) with deep learning techniques. Current methods encounter challenges such as low accuracy, limited efficiency, and constrained application scenarios. We introduce an innovative approach that employs a UAV equipped with a binocular camera for identifying pavement surface cracks. This method is augmented by a binocular ranging algorithm combined with edge detection and skeleton extraction algorithms, enabling the quantification of crack widths without necessitating a preset shooting distance—a notable limitation in existing UAV crack detection applications. We developed an optimized model to enhance detection accuracy, incorporating the YOLOv5s network with an Efficient Channel Attention (ECA) mechanism. This model features a decoupled head structure, replacing the original coupled head structure to optimize detection performance, and utilizes a Generalized Intersection over Union (GIoU) loss function for refined bounding box predictions. Post identification, images within the bounding boxes are segmented by the Unet++ network to accurately quantify cracks. The efficacy of the proposed method was validated on roads in complex environments, achieving a mean Average Precision (mAP) of 86.32% for crack identification and localization with the improved model. This represents a 5.30% increase in the mAP and a 6.25% increase in recall compared to the baseline network. Quantitative results indicate that the measurement error margin for crack widths was 10%, fulfilling the practical requirements for pavement crack quantification.

Optimization Method for Robot Moving Object Recognition and Grasping Strategy Based on Binocular Vision

<p>This article proposes a more accurate grasping strategy for the recognition and grasping of moving targets based on binocular vision cameras. Firstly, the front and back scene separation algorithm is used to identify the moving target grabbing object in the production line. Then, by setting an appropriate threshold, the SiamMask target tracking algorithm is improved to achieve dynamic target tracking. Finally, the conveyor belt speed is detected and the real-time position of the object is obtained. Then, the Cartesian strategy is used to achieve path planning and optimization methods for the robotic arm during movement. Through experimental simulation, the effectiveness and stability of the proposed method in this paper have been demonstrated.</p> <p> </p>

A target-free vision-based method for out-of-plane vibration measurement using projection speckle and camera self-calibration technology

In recent decades, computer vision-based methods have been introduced into engineering structures to provide full-field, non-contact vibration measurement. However, the vision-based vibration measurement method is challenging to measure the targetless structures. This paper proposed a target-free out-of-plane vibration measurement method using projection speckle and camera self-calibration technology. In this method, the camera extrinsic parameter estimation result based on the direct method is used as the initial value in the proposed method. Then, a nonlinear inequality constraint on the extrinsic parameters is established regarding the epipolar condition to minimize the reprojection error. Furthermore, the projection relationship between the projection speckle displacement and the actual displacement of the structure is studied so that the deviation of the out-of-plane vibration displacement is corrected. In the experimental verification, 3D reconstruction and rigid translation experiments are designed, respectively. The results show that the self-calibration accuracy of the binocular camera proposed in this paper is notably improved compared with the existing methods. At the same time, the deviation correction effect of the vibration displacement is significant. Finally, the typical flat plate structure and composite wing structure are taken as vibration measurement cases to demonstrate that the proposed approach is reliable and accurate in measuring the dynamic responses of engineering structures.

Design of 3D Environment Combining Digital Image Processing Technology and Convolutional Neural Network

As virtual reality technology advances, 3D environment design and modeling have garnered increasing attention. Applications in networked virtual environments span urban planning, industrial design, and manufacturing, among other fields. However, existing 3D modeling methods exhibit high reconstruction error precision, limiting their practicality in many domains, particularly environmental design. To enhance 3D reconstruction accuracy, this study proposes a digital image processing technology that combines binocular camera calibration, stereo correction, and a convolutional neural network (CNN) algorithm for optimization and improvement. By employing the refined stereo-matching algorithm, a 3D reconstruction model was developed to augment 3D environment design and reconstruction accuracy while optimizing the 3D reconstruction effect. An experiment using the ShapeNet dataset demonstrated that the evaluation indices—Chamfer distance (CD), Earth mover’s distance (EMD), and intersection over union—of the model constructed in this study outperformed those of alternative methods. After incorporating the CNN module in the ablation experiment, CD and EMD increased by an average of 0.1 and 0.06, respectively. This validates that the proposed CNN module effectively enhances point cloud reconstruction accuracy. Upon adding the CNN module, the CD index and EMD index in the dataset increased by an average of 0.34 and 0.54, respectively. These results indicate that the proposed CNN module exhibits strong predictive capabilities for point cloud coordinates. Furthermore, the model demonstrates good generalization performance.

Cluster optimization and algorithm design for machine vision in industrial robot control systems

Abstract Machine vision technology improves the ability to detect the environment of industrial robots, which contributes to the improvement of the collaboration efficiency of swarm robots. This paper proposes a roadmap extraction algorithm that utilizes the improved Mean Shift algorithm to extract the roadmap information from images acquired by binocular cameras. Subsequently, the IEO algorithm with K-Means++ is used to optimize the task allocation of the swarm robots. The experiments show that the average error of this paper’s algorithm’s road sign extraction is 0.025m, the ratio of full scene and homing reaches 90.6%, and the results of the scheduling algorithm under the three kinds of task volume are 59.89, 773.08, and 2704.67. The efficiency of scheduling task completion in dispensing experiments is 9.56% higher than that of the comparative algorithms. The experiment proves that the algorithm proposed in this paper has good performance and practical effects on optimizing the industrial robot control system.

A CNN-Based Online Self-Calibration of Binocular Stereo Cameras for Pose Change

This paper proposes a novel method that can automatically and accurately recognize the pose change of binocular stereo cameras in real time and correct these changes. Focused on predicting a five degree-of-freedom extrinsic pose, we design a convolutional neural network (CNN) that implements the regression of rotation angles of two cameras. The proposed method increases regression accuracy using the information inherent in the entire image. To this end, the CNN divides the image into patches of a certain size, extracts detailed features and context features of the patches, and extracts attention information for patches belonging to the left and right images. Training and evaluating the CNN requires many stereo images with variations from the initial setup of the cameras. We solve this problem using miscalibration. In miscalibration, angles expected to be rotated for the three axes of the left and right cameras are randomly sampled within a range of ±2.5°, and a pair of rectified images are transformed using the sampled angles. The CNN uses these transformed images to infer the angle at which the camera axis is expected to have been rotated. Then, the pair of transformed images are corrected with these inferred angles. The superiority of the proposed method is demonstrated using the KITTI odometry dataset and the GY dataset we built.