Binocular Camera Research Articles

Auto calibration of multi‐camera system for human pose estimation

The multi-camera calibration is an essential step for many spatially aware applications, such as robotic navigation, augmented reality, and 3D human pose estimation. Traditional calibration methods use off-the-shelf checkerboards or triangles as the known world coordinate system and their corresponding corners are set as control points, which heavily depends on specific calibration patterns and is not suitable for calibration pattern-denied environments. In this paper, an automatic calibration method is proposed to calibrate the multi-camera system without the aid of a known calibration pattern. The key idea of the proposed method is that the authors consider the human body, which is always available, as the counterpart of the calibration pattern. The authors’ approach starts with binocular camera calibration, in which the extrinsic and intrinsic parameters are calculated in order and followed by a joint optimisation. With the results of each pair of binocular camera calibration, the multi-camera system calibration is carried out in three steps: (i) parameters initialisation, (ii) extrinsic parameters optimisation, and (iii) jointly optimising intrinsic and extrinsic parameters. Since the authors’ approach does not require additional calibration patterns except for one visible person, it is flexible and easy to be implemented. Real experiments are conducted in different scenes, camera angles, and camera settings. Human pose estimation with the multi-camera system is additionally performed for exhaustive experiments. The experimental results demonstrate that the authors’ method shows superior performance than the traditional method with the aid of a specific calibration pattern.

Real and Pseudo Pedestrian Detection Method with CA-YOLOv5s Based on Stereo Image Fusion.

With the development of convolutional neural networks, the effect of pedestrian detection has been greatly improved by deep learning models. However, the presence of pseudo pedestrians will lead to accuracy reduction in pedestrian detection. To solve the problem that the existing pedestrian detection algorithms cannot distinguish pseudo pedestrians from real pedestrians, a real and pseudo pedestrian detection method with CA-YOLOv5s based on stereo image fusion is proposed in this paper. Firstly, the two-view images of the pedestrian are captured by a binocular stereo camera. Then, a proposed CA-YOLOv5s pedestrian detection algorithm is used for the left-view and right-view images, respectively, to detect the respective pedestrian regions. Afterwards, the detected left-view and right-view pedestrian regions are matched to obtain the feature point set, and the 3D spatial coordinates of the feature point set are calculated with Zhengyou Zhang’s calibration method. Finally, the RANSAC plane-fitting algorithm is adopted to extract the 3D features of the feature point set, and the real and pseudo pedestrian detection is achieved by the trained SVM. The proposed real and pseudo pedestrian detection method with CA-YOLOv5s based on stereo image fusion effectively solves the pseudo pedestrian detection problem and efficiently improves the accuracy. Experimental results also show that for the dataset with real and pseudo pedestrians, the proposed method significantly outperforms other existing pedestrian detection algorithms in terms of accuracy and precision.

Design of binocular stereo vision optical system based on a single lens and a single sensor.

To reduce the number of detectors used in conventional binocular stereo cameras, while improving the measurement accuracy and compactness of the system, this paper proposes a design method for a binocular stereo vision optical system based on a single lens and a single sensor. First, based on the design principle of the traditional binocular optical system, to the best of our knowledge, a novel method of designing a framing lens array at the optical stop of the optical system is proposed to image two images on one detector simultaneously. Second, we propose a dual-frame lens array design method at the aperture stop position of the optical system that can image two images on one detector simultaneously. Then, the design principle of the method is analyzed theoretically, as well as a detailed analysis of the imaging position layout and the stray light elimination method of the dual-channel optical system. Finally, a single-lens binocular optical system with a focal length of 20 mm and a full field of view of 30° is designed using the method in this paper, and the analysis results demonstrate that the system has the advantages of good imaging quality and compact construction and provides a design idea for the design of a binocular stereo vision optical system.

Image Recognition of Sports Athletes' High-Intensity Sports Injuries Based on Binocular Stereo Vision.

Sports athletes are prone to certain injuries during high-intensity exercise training. In the process of treating an injury, images of the injury site need to be collected and identified. However, the traditional recognition method cannot effectively extract the features of the image. At the same time, it ignores the optimization of the damage image recognition results, resulting in low recognition accuracy and poor efficiency. Binocular stereo vision technology can quickly and accurately detect moving objects. Therefore, in order to more accurately identify high-intensity sports injury images, this study takes the high-intensity sports injury images as the basic research object. Several processes of image processing based on binocular stereo vision are analyzed, and the vulnerable parts of the body in high-intensity sports are also studied. Finally, the method in this study is verified. The experimental results show that the method proposed in this study reduces the average error rate by 0.19% compared with the traditional recognition method. It can effectively identify and detect injury images, thereby improving the accuracy and stability of sports injury image identification. The identification time is also shortened accordingly, which has certain practicability and feasibility. In addition, the binocular camera used in this study has high accuracy, and the obtained images of sports injuries are of good quality, which lays a foundation for image detection and recognition.

Anatomical mining method of cervical nerve root syndrome under visual sensing technology

INTRODUCTION: The gray resolution of anatomical image of cervical nerve root syndrome is low, that can not be mined accurately. OBJECTIVES: Aiming at the defect of low gray resolution of anatomical images, an image mining method using visual perception technology was studied. METHODS: According to the visual perception technology, the internal parameter matrix and external parameter matrix of binocular visual camera were determined by coordinate transformation, and the anatomical images of cervical nerve root syndrome were collected. The collected images are smoothed and enhanced by nonlinear smoothing algorithm and multi-scale nonlinear contrast enhancement method. The directional binary simple descriptor method is selected to extract the features of the enhanced image; Using K-means clustering algorithm, the anatomical image mining of cervical nerve root syndrome is completed by obtaining the initial clustering center and image mining. RESULTS: Experimental results show that the information entropy of the images mined by the proposed method is higher than 5, the average gradient is greater than 7, the edge information retention is greater than 0.7, the peak signal-to-noise ratio is higher than 30 dB, and the similarity of the same category of images is greater than 0.9. CONCLUSIONS: This method can effectively mine the anatomical images of cervical nerve root syndrome and provide an important basis for the diagnosis and treatment of cervical nerve root syndrome.

Detecting coal-carrying rate in gangue based on binocular machine vision and particle queuing method

ABSTRACT To solve the problem of online detection of coal-carrying rate in gangue, an approach based on particle queuing and binocular machine vision was proposed. First, a queuing device was applied to line up the coal and gangue particles. A binocular camera was used to obtain images of coal and gangue from above the belt. Through image segmentation, the coal and gangue particle regions were divided from the belt background. Then, the Semi-Global Block Matching (SGBM) algorithm was used to obtain the height information matrix of each region. At the same time, the projected area of the particles was calculated from the area of the divided regions. It was converted from pixel units to physical units by capturing an image of a ruler. The height matrix and projected area were combined to construct a mechanism-based volume model by integral principle. Finally, the calculated volume of coal and gangue were multiplied by their respective empirical density to get the mass of particles. The coal-carrying rate in gangue was obtained by dividing the mass of coal by the total mass. The average relative error of the proposed method is 2.79%. It not only overcomes the prediction error caused by particle stacking, but also improves the computational efficiency of image processing, and the “white-box model” is also easy to understand.

Improving 2D object detection with binocular images for outdoor surveillance

Detecting objects and providing their 2D information (e.g., size and center) are crucial for outdoor visual surveillance. Because the cameras are static and their distances to background are fixed in the scenario of surveillance, we argue that, compared to 3D object detection, 2D object detection is usually enough in visual surveillance. To a great extent, determining the existence of objects is more important than determining their precise 3D shapes and locations. In the field of visual surveillance, almost all methods employ a single camera (or independently employing several cameras) to determine the existence and 2D information of objects. In this paper, we propose to improve the performance of the 2D object detection by binocular cameras (images) for the scenario of outdoor surveillance where RGB-D and depth cameras are not applicable due to their low resolutions and high expense. In the proposed binocular-image based 2D object detection framework, most of existing monocular-image based 2D object detection modules can be integrated in. Once integrated in the proposed framework, the object detection accuracy can significantly and consistently be improved. Experimental results on the challenging datasets of Cityscapes and KITTI demonstrate the superiority of the proposed method.

Neural-Network-Based Model-Free Calibration Method for Stereo Fisheye Camera.

The fisheye camera has a field of view (FOV) of over 180°, which has advantages in the fields of medicine and precision measurement. Ordinary pinhole models have difficulty in fitting the severe barrel distortion of the fisheye camera. Therefore, it is necessary to apply a nonlinear geometric model to model this distortion in measurement applications, while the process is computationally complex. To solve the problem, this paper proposes a model-free stereo calibration method for binocular fisheye camera based on neural-network. The neural-network can implicitly describe the nonlinear mapping relationship between image and spatial coordinates in the scene. We use a feature extraction method based on three-step phase-shift method. Compared with the conventional stereo calibration of fisheye cameras, our method does not require image correction and matching. The spatial coordinates of the points in the common field of view of binocular fisheye camera can all be calculated by the generalized fitting capability of the neural-network. Our method preserves the advantage of the broad field of view of the fisheye camera. The experimental results show that our method is more suitable for fisheye cameras with significant distortion.

Simplified 4-DOF manipulator for rapid robotic apple harvesting

Fruit harvesting is time-consuming and laborious. Robots can harvest fruits with a high degree of automation, greatly reducing labor requirements, but are limited by their efficiency and high costs. In this research, a prototype apple harvesting robot was designed and constructed. The robotic prototype integrated a binocular camera, a 4-degree-of-freedom manipulator, a vacuum-based end-effector, and a mobile vehicle. The robot detected, positioned, gripped, detached, and placed apples. A manipulator controller was designed to realize rapid control execution. Picking experiments were conducted in a spindle apple orchard. Picking was tested using rotation-pull and pull patterns. The test results showed that the rotation-pull pattern was more effective picking apples than the pull pattern. The picking success rate of the rotation-pull pattern was 47.37% in the field orchard and 78% in the simulated orchard environment, with picking cycle time of ∼ 4 s. The stem damage rate in the field orchard was 11.11%. The developed picking prototype realized the task of picking apples. Its primary advantage was its competitive picking cycle time, which provides a solid foundation for the future improvement of harvesting efficiency.

Monitoring and automatic characterization of cracks in strain-hardening cementitious composite (SHCC) through intelligent interpretation of photos

This paper presents an intelligent photo interpretation approach to automatically monitor and characterize dense interconnected microcracks in strain-hardening cementitious composite (SHCC) featuring unique crack patterns in terms of crack number and crack width. The presented approach employs a stereo vision system that integrates binocular and monocular cameras for automatic detection, ranging, and quantification of cracks as well as characterization of crack patterns. The presented approach was implemented into evaluation of SHCC in flexural tests and direct tension tests. Dense microcracks were detected and ranged by the stereo vision system, segmented by an encoder-decoder approach, and quantified by an efficient computer vision approach. Evolution of the cracks was traced throughout the loading process until failure, and a statistical analysis revealed that the crack width was retained while the crack number monotonically increased. The interpretation time was shorter than 0.4 s for each photo, making the approach promising for monitoring of SHCC. The proposed system can be deployed for automated assessment of cementitious composites with complex crack patterns in material research and engineering structures.

Multi-Sensor Fusion for Lateral Vehicle Localization in Tunnels

The satellite navigation signal in the tunnel is weak, and it is difficult to achieve accurate lateral positioning in complex conditions such as low-speed congestion by relying solely on inertial navigation or line image recognition, which is one of the problems of automatic driving at present. In this paper, a lane-level location method based on multi-sensor fusion is proposed. Using the machine vision method, detecting lane lines with the monocular camera, and fitting the lane lines to determine the driving status of the vehicle based on the lane line information. The top view of the lane line is taken by the binocular camera, and the distance of the vehicle from the lane line and the width of the lane are calculated from the pictures taken by the binocular camera. Obtaining the heading angle information of the vehicle using the gyroscope in inertial navigation and the distance information of the vehicle using the odometer. When a car changes lanes or overtakes, the new lane the vehicle is in is calculated by calculating the difference in heading angle and combining it with the lane width and odometer information so as to complete the lateral positioning of the vehicle. The simulation results show that the algorithm has high lateral positioning accuracy. The positioning accuracy is less affected by the drift of inertial elements, and the error will not accumulate.

Image Principal Distance Calibration Method by Binocular Camera Based on Improved Differential Evolution Algorithm

The traditional camera calibration method usually relies on the calibration plate, mainly uses the exhaustive method, and converts the spatial coordinates into image plane coordinates by solving the resection of the equation. The accuracy and calculation rate of traditional algorithms can no longer fully meet the needs of engineering applications. In view of this situation, this paper proposes an improved differential evolution algorithm (IDE) method for binocular camera calibration. The differential evolution algorithm (DE) is an algorithm that simulates the biological evolution process and uses the survival of the fittest to gradually screen individuals to find the optimal solution. The standard DE algorithm needs to perform population initialization, mutation operation and crossover operation, and finally screen individuals through greedy algorithm. In the mutation operation of the DE algorithm, the coefficient of variation F plays a crucial role in the result but it is difficult to obtain a value. Therefore, it is necessary to optimize the operator of the coefficient of variation F, so that the coefficient of variation F also reaches the optimal solution, which is the IDE algorithm. In this paper, the traditional calculation method, the standard DE algorithm and the IDE algorithm are used to calibrate the principal distance of the two images, and the accuracy and iteration rate are compared. The experimental results show that the IDE algorithm has obvious advantages in terms of accuracy, especially the calculation speed.

Possible Ongoing Merger Discovered by Photometry and Spectroscopy in the Field of the Galaxy Cluster PLCK G165.7+67.0

We present a detailed study of the Planck-selected binary galaxy cluster PLCK G165.7+67.0 (G165; z = 0.348). A multiband photometric catalog is generated incorporating new imaging from the Large Binocular Telescope/Large Binocular Camera and Spitzer/IRAC to existing imaging. To cope with the different image characteristics, robust methods are applied in the extraction of the matched-aperture photometry. Photometric redshifts are estimated for 143 galaxies in the 4 arcmin2 field of overlap covered by these data. We confirm that strong-lensing effects yield 30 images of 11 background galaxies, of which we contribute new photometric redshift estimates for three image multiplicities. These constraints enable the construction of a revised lens model with a total mass of M 600 kpc = (2.36 ± 0.23) × 1014 M ⊙. In parallel, new spectroscopy using MMT/Binospec and archival data contributes thirteen galaxies that meet our velocity and transverse radius criteria for cluster membership. The two cluster components have a pair-wise velocity of ≲100 km s−1, favoring an orientation in the plane of the sky with a transverse velocity of 100–1700 km s−1. At the same time, the brightest cluster galaxy (BCG) is offset in velocity from the systemic mean value, suggesting dynamical disturbance. New LOFAR and Very Large Array data uncover head-tail radio galaxies in the BCG and a large red galaxy in the northeast component. From the orientation and alignment of the four radio trails, we infer that the two cluster components have already traversed each other, and are now exiting the cluster.

Asynchronous Double-Frame-Exposure Binocular-Camera With Pixel-Level Pipeline Architecture for High-Speed Motion Tracking

The synchronous double-frame exposure (S-DFE), also called frame-straddling, has been widely applied for commercial video-based velocity measurements to provide high-speed velocimetry-flow images. Nevertheless, no low-cost, high-speed cameras offer the video quality that researchers, scientists, and industry demand. This brief presents an asynchronous DFE (A-DFE) method with a binocular camera for motion estimation, which captures double frames with the scalable time interval of the dual generic cameras. This time interval can significantly improve the upper limit of displacements of the motion, which is commonly solved by an expensive high-speed camera. A dedicated hardware architecture is designed to implement the perspective transformation and Wallis filter to eliminate the two cameras’ position mismatch and brightness variance. We build a high-speed Lucas-Kanade optical flow estimation system with A-DFE on the XILINX Spartan-7 FPGA platform with MT9V034 CMOS image sensors. The shortest time interval can reach 1.06 ns, which means a significant improvement of the upper limit of the measurement velocity. Except for the image sensor, the compact, fully parallel architecture can attain energy efficiency with 3.23 nJ/pixel on a low-cost FPGA device.

UAV Landing Aid Hexapod Robot based on ArUco Marker and Sparse Optical Flow

A UAV (Unmanned Aerial Vehicle) landing robot system based on a six-degree-of-freedom hexapod and computer vision with ArUco marker and sparse optical flow is implemented in this paper. By using binocular camera measuring the real-time position and orientation of UAV, a dynamic target tracking method based on ArUco marker and sparse optical flow is proposed. In this method, the ArUco marker is combined with sparse optical flow to improve the accuracy of target detection and the matching error of binocular stereo matching. The binocular vision system with ArUco marker is used to measure and calculate the relative coordinates of markers in the camera coordinate system. Then the UAV contour fitting is conducted in order to obtain the real-time position and orientation of this mobile target. The effectiveness of the novel UAV landing hexapod robot system based on ArUco marker combined with sparse optical flow is verified through the UAV capture experiment.

A depth-colour image registration method based on local feature point extraction

A binocular camera composed of colour and depth cameras is an accurate and inexpensive tool for recognising fruit trees to plan spraying paths. For this purpose, however, the accurate visual registration of the depth and colour cameras is a problem that must be solved. This paper proposes a local feature point extraction (LFPE) algorithm for depth-colour image registration, which automatically extracts the feature points of local regions in the colour and depth images for registration. First, the binarised colour and depth images are divided into local regions after edge extraction, and the regions in which the edge features match at several pixels between images are selected. Then, these regions are again divided into smaller regions to detect local extrema of the pixels as feature points. Finally, the matching feature points are calculated based on the Euclidean distance and are used to optimise the affine transformation matrix. To verify the accuracy and effectiveness of the algorithm, it is compared with the FAST, SURF and BRISK algorithms. The experimental results show that the accuracy of the proposed LFPE algorithm is 1.6 times that of BRISK, 3.5 times that of FAST and 40.1 times that of SURF. Therefore, it is more suitable for registration between depth and colour images. • Proposed a depth-colour image registration method for orchard precision spraying. • A local feature point extraction (LFPE) algorithm is proposed to match images. • The registration accuracy of LFPE is much better than other similar algorithms.

Modal analysis and damage localization in plate-type structures via TDD and PE methods based on the data of an integrated highspeed camera system

With the fast development of full-field vibration measurements by combining digital image correlation (DIC) technique with the integrated highspeed camera system, the structural characteristic deflection shapes (CDS’s) can be readily captured, which are sensitive for damage localization. Naturally, the mode shapes are preferred due to their higher signal to noise ratio. However, traditional modal analysis methods are inefficient in processing the large volumes of data that acquired by high-speed camera system for mode shape-based damage localization. Moreover, how to extract the damage features and enhance the damage localization results without the baseline-data of the intact state is another challenging issue. To address these problems, a novel damage localization method is proposed by enhancing the mode shape estimation and optimizing the damage feature extraction. Firstly, the dominant singular vector at each interested resonant frequency is individually evaluated as the mode shapes in the proposed time domain decomposition (TDD) method. Moreover, based on the extracted mode shapes, a pseudo-excitation (PE) method is proposed to reveal the damage locations without baseline-data by examining the perturbation of local dynamic equilibrium. To circumvent the effects of measurement noise on PE method, an adaptive denoising technique is proposed based on Gaussian smoothing. Furthermore, the local contiguity and spatial sparsity of the damage-caused features are harnessed by a hierarchical clustering to optimize the damage localization performance via tuning the scale parameter of Gaussian smoothing. Finally, numerical and experimental studies of crack damaged plates are conducted to validate the feasibility and effectiveness of the proposed mode shape estimation and damage localization methods. During the experiment, the full-field vibration measurements are acquired by using an integrated binocular highspeed camera system. In addition, a comparison study based on benchmark data is carried out to demonstrate the performance and reliability of the proposed damage localization method.

3D Locating System for Pests’ Laser Control Based on Multi-Constraint Stereo Matching

To achieve pest elimination on leaves with laser power, it is essential to locate the laser strike point on the pest accurately. In this paper, Pieris rapae (L.) (Lepidoptera: Pieridae), similar in color to the host plant, was taken as the object and the method for identifying and locating the target point was researched. A binocular camera unit with an optical filter of 850 nm wavelength was designed to capture the pest image. The segmentation of the pests’ pixel area was performed based on Mask R-CNN. The laser strike points were located by extracting the skeleton through an improved ZS thinning algorithm. To obtain the 3D coordinates of the target point precisely, a multi-constrained matching method was adopted on the stereo rectification images and the subpixel target points in the images on the left and right were optimally matched through fitting the optimal parallax value. As the results of the field test showed, the average precision of the ResNet50-based Mask R-CNN was 94.24%. The maximum errors in the X-axis, the Y-axis, and the Z-axis were 0.98, 0.68, and 1.16 mm, respectively, when the working depth ranged between 400 and 600 mm. The research was supposed to provide technical support for robotic pest control in vegetables.

Research on the Industrial Robot Grasping Method Based on Multisensor Data Fusion and Binocular Vision.

At present, most of the handling industrial robots working on the production line are operated by teaching or preprogramming, which makes the flexibility of the production line poor and does not meet the flexible production requirements of the material handling system. This study proposes a solution based on adding computer binocular vision to a five-axis industrial robot system. A simple and high-precision binocular camera calibration method is proposed, the kinematics of the five-axis robot is analyzed, and the target positioning is realized; the communication between the upper and lower robots is realized through Ethernet. According to the specific target, the grasping scheme of the gripper was designed; the control software was developed using two schemes. Visual control is carried out by operating specific buttons on the control panel, and visual control is carried out by executing the macrovariable program, finally realizing the joint fusion of multisensor data and binocular vision.

Measurement of Large-Sized-Pipe Diameter Based on Stereo Vision

To address the lack of fast and high-precision in situ measurement of large-sized pipes in current industrial applications, a pipe-diameter-measurement method based on stereo vision is designed in this paper. By using multiple sets of binocular cameras to perform 3D reconstruction and integration of multiple laser markers projected on measured cross sections of pipes, the pipe diameter can be estimated. In this method, a measurement algorithm is adopted to enable automatic matching of feature points through affine distance transformation, and an optimized point-cloud-registration algorithm with normal-vector constraints is used to ensure measurement robustness. To verify the feasibility of the method, an experimental system was built under laboratory conditions, and three types of pipes with outer diameters from 285 mm to 325 mm were measured. The experimental results show that the relative error is within ±0.570% and the maximum repeatability standard deviation is 0.551 mm. The experimental results basically meet industrial standards, and the proposed method therefore has good application prospects.