Semi-global Matching Algorithm Research Articles

Research on image deformation monitoring algorithm based on binocular vision

Monitoring the deformation of critical structures such as tunnel、buildings and bridges is critical to ensure the safety of infrastructures. Non-contact deformation monitoring technology based on digital imagery has emerged as a prevalent and effective monitoring method, which enables cost-effective and efficient tracking of structural deformations. This paper aims to develop an automated deformation monitoring methodology using the binocular vision. First, standard preprocessing of digital images is performed through the integration of filtering algorithms and image enhancement techniques. Furthermore, by combining the Scale Invariant Feature Transform (SIFT) feature matching algorithm, the inverse compositional gauss–newton iterative algorithm, and interpolation methodologies, the planar displacement of the monitored targets can be calculated precisely, with sub-pixel precision being achieved. Subsequently, by adopting an enhanced Semi-Global Matching (SGM) algorithm, stereo matching of binocular images were accomplished. This process yields a disparity map, from which the depth map is computed. Then, precise measurement of the vertical displacement of the monitoring target can be obtained. This enables accurate measurement of three-dimensional displacements of the monitored targets. Finally, the developed methodology undergoes a comparative assessment against traditional measurement methods and existing algorithms to validate its measurement accuracy. The results prove that the deformation measurement technique based on binocular vision proposed in this paper captures structural deformation effectively. The accuracy surpasses that of existing measurement algorithms, and its efficiency outperforms traditional measurement methods.

A Hardware Accelerator for the Semi-Global Matching Stereo Algorithm: An Efficient Implementation for the Stratix V and Zynq UltraScale+ FPGA Technology

The semi-global matching stereo algorithm is a top performing algorithm in stereo vision. The recursive nature of the computations involved in this algorithm introduces an inherent data dependency problem, hindering the progressive computations of disparities at pixel clock. In this work, a novel hardware implementation of the semi-global matching algorithm is presented. A hardware structure of parallel comparators is proposed for the fast computation of the minima among large cost arrays in one clock cycle. Also, a hardware-friendly algorithm is proposed for the computation of the minima among far-indexed disparity costs, shortening the length of computations in the datapath. As a result, the recursive path cost computation is accelerated considerably. The system is implemented in a Stratix V device and in a Zynq UltraScale+ device. A throughput of 55,1 million disparities per second is achieved with maximum disparity 128 pixels and frame resolution 1280 × 720. The proposed architecture is less elaborate and more resource efficient than other systems in the literature and its performance compares favorably to them. An implementation on an actual FPGA board is also presented and serves as a real-world verification of the proposed system.

Three-dimensional reconstruction optimization of tunnel face and intelligent extraction of discontinuity orientation based on binocular stereo vision

In the process of grading and dynamically optimizing the design and construction parameters of the surrounding rock mass of a rock tunnel face, efficiently and accurately acquiring the geometrical parameters of the rock discontinuities is an important basic task. To address the problems of time consuming, low accuracy, and high danger associated with traditional methods of obtaining the structural information of rock mass, this paper proposes a method for three-dimensional reconstruction and intelligent information extraction of tunnel face based on binocular stereo vision (BSV). First, the parallel binocular device with a single camera was improved, calibrated using the checkerboard calibration method. By integrating with the semi-global matching algorithm, the BSV based method for the three-dimensional reconstruction of the rock mass of the tunnel face was optimized. Furthermore, based on the results from on-site engineering applications, this study leveraged two parameters, point cloud density and algorithm runtime, to determine the optimal values for the disparity range and window size parameters within the semi-global stereo matching algorithm. This enhancement improved the performance of the 3D reconstruction method based on binocular stereo vision. Finally, efficient and refined intelligent methods for extracting structural parameters of the rock mass were proposed based on k-nearest neighbor search and kernel density estimation. The research results can provide reliable technical support for the intelligent and efficient acquisition of rock mass structural information in rock tunnel engineering faces.

CONTINUOUS 3D-LABEL SEMI-GLOBAL MATCHING FOR SATELLITE STEREO

Abstract. We present Continuous 3D-Label Semi-Global Matching (CoSGM), a new dense stereo matching algorithm for satellite stereo. CoSGM overparameterizes the disparity map as an array of local planes, so it can regularize the first-order smoothness of the estimated disparity. Furthermore, the algorithm can produce a dense normal map beside the disparity map. We show experimentally that CoSGM could achieve denser depth maps with comparable accuracy to traditional semi-global matching algorithms.

ArthroNavi framework: stereo endoscope-guided instrument localization for arthroscopic minimally invasive surgeries.

As an example of a minimally invasive arthroscopic surgical procedure, arthroscopic osteochondral autograft transplantation (OAT) is a common option for repairing focal cartilage defects in the knee joints. Arthroscopic OAT offers considerable benefits to patients, such as less post-operative pain and shorter hospital stays. However, performing OAT arthroscopically is an extremely demanding task because the osteochondral graft harvester must remain perpendicular to the cartilage surface to avoid differences in angulation. We present a practical ArthroNavi framework for instrument pose localization by combining a self-developed stereo endoscopy with electromagnetic computation, which equips surgeons with surgical navigation assistance that eases the operational constraints of arthroscopic OAT surgery. A prototype of a stereo endoscope specifically fit for a texture-less scene is introduced extensively. Then, the proposed framework employs the semi-global matching algorithm integrating the matching cubes method for real-time processing of the 3D point cloud. To address issues regarding initialization and occlusion, a displaying method based on patient tracking coordinates is proposed for intra-operative robust navigation. A geometrical constraint method that utilizes the 3D point cloud is used to compute a pose for the instrument. Finally, a hemisphere tabulation method is presented for pose accuracy evaluation. Experimental results show that our endoscope achieves 3D shape measurement with an accuracy of . The mean error of pose localization is 15.4deg (range of 10.3deg to 21.3deg; standard deviation of 3.08deg) in our ArthroNavi method, which is within the same order of magnitude as that achieved by experienced surgeons using a freehand technique. The effectiveness of the proposed ArthroNavi has been validated on a phantom femur. The potential contribution of this framework may provide a new computer-aided option for arthroscopic OAT surgery.

Initializing and accelerating Stereo-DIC computation using semi-global matching with geometric constraints

Stereo Digital Image Correlation (Stereo-DIC) has become a mainstream optical metrology technique for quantitatively analyzing full-field 3D shape, displacement, or deformation of materials and structures. Whether it is to measure 3D profile or deformation, stereo matching is essential for Stereo-DIC to reconstruct 3D point clouds from stereo images. Traditional feature-based (e.g., SIFT) methods provide initial 2D displacements for stereo matching with the aid of extracting massive features, but at the cost of expensive computational overhead. In addition, these methods preclude precise measurement of objects with steep and ridged surfaces or undergoing large rotation and/or deformation due to low feature matching accuracy of complex regions caused by perspective differences. In this paper, we propose a fast and robust stereo matching method using semi-global matching with geometric constraints (GC-SGM) for initializing and accelerating Stereo-DIC computation. For GC-SGM, an optimized semi-global matching (SGM) algorithm based on GPU acceleration is first utilized to quickly estimate dense and reliable disparity maps between the rectified stereo images. The global pixel-wise 2D correspondence between raw stereo images can be established inversely using epipolar constraints and 1D disparity information, and then converted to accurate and initial second-order deformation parameters for 2D-DIC-based sub-pixel refinement by least-squares-based surface fitting. Experimental results prove that the proposed GC-SGM enhances the matching correctness and robustness for complex objects while improving the processing speed on GPU by 3∼10 times compared with SIFT-based methods, enabling high-precision and computationally efficient 3D shape and deformation measurement.

Leveraging Chinese GaoFen-7 imagery for high-resolution building height estimation in multiple cities

Accurate building heights are essential to understand the mechanisms of urban systems and provide new insights into research on infrastructure equality, urban environment, population modeling, etc. However, relatively few studies have been conducted to estimate high-resolution building heights using Chinese GaoFen-7 imagery across multiple cities. In this study, we developed an effective Building Height Estimation method by synergizing Photogrammetry and Deep learning methods (BHEPD) to leverage Chinese multi-view GaoFen-7 (GF-7) images for high-resolution building height estimation. In particular, BHEPD incorporates the StereoNet model and is capable of generating an edge-preserving digital surface model (DSM) from GF-7 images. This improvement enhances the accuracy of heights estimation, particularly for high-rise, high-density, and multi-scale buildings. BHEPD extracted over 700,000 building heights in local areas of Tianjin, Lanzhou, Chongqing, Ningbo, Guangzhou, Foshan, Shenzhen, and Hong Kong. The estimated building heights in the different cities are highly consistent with the reference building height data obtained from field-measured building height data, building floor data, and airborne light detection and ranging (LiDAR) data. The R2 varies from 0.75 to 0.90, while the root mean square error (RMSE) ranges from 4.06 m to 8.06 m. The validation of the estimated building heights over 106,366 buildings using airborne LiDAR height data yielded R2 and RMSE of 0.90 and 6.72 m for Shenzhen and 0.89 and 4.06 m for Hong Kong, respectively. Furthermore, a comparison with the building height estimation based on the traditional DSM generation method (i.e., semi-global matching (SGM) algorithm) confirmed the superiority of BHEPD in alleviating the underestimation of high-rise buildings (>60 m). Overall, BHEPD shows great potential for high-resolution building height estimation and can be expected to reduce the manual workload involved in collecting building height information on large scale.

Research on the Improvement of Semi-Global Matching Algorithm for Binocular Vision Based on Lunar Surface Environment.

The low light conditions, abundant dust, and rocky terrain on the lunar surface pose challenges for scientific research. To effectively perceive the surrounding environment, lunar rovers are equipped with binocular cameras. In this paper, with the aim of accurately detect obstacles on the lunar surface under complex conditions, an Improved Semi-Global Matching (I-SGM) algorithm for the binocular cameras is proposed. The proposed method first carries out a cost calculation based on the improved Census transform and an adaptive window based on a connected component. Then, cost aggregation is performed using cross-based cost aggregation in the AD-Census algorithm and the initial disparity of the image is calculated via the Winner-Takes-All (WTA) strategy. Finally, disparity optimization is performed using left-right consistency detection and disparity padding. Utilizing standard test image pairs provided by the Middleburry website, the results of the test reveal that the algorithm can effectively improve the matching accuracy of the SGM algorithm, while reducing the running time of the program and enhancing noise immunity. Furthermore, when applying the I-SGM algorithm to the simulated lunar environment, the results show that the I-SGM algorithm is applicable in dim conditions on the lunar surface and can better help a lunar rover to detect obstacles during its travel.

An Autonomous Surgical Instrument Tracking Framework With a Binocular Camera for a Robotic Flexible Laparoscope

In minimally invasive surgery (MIS), the field of view (FOV) plays a vital role. To enhance the stability of FOV and lighten the burden on surgeons, robot-assisted laparoscope systems have been developed and introduced into surgery. However, most of the existing automatic surgical tool tracking schemes with robotic laparoscopes either do not consider tracking of the depth direction or the remote center of motion (RCM) constraints. To tackle these problems, this paper presents an autonomous three-dimensional surgical instrument tracking framework for a robotic laparoscope based on a binocular camera. A 3-degree-of-freedom visual tracking scheme is derived from the kinematics of the binocular robotic laparoscope. A programmable RCM constraint is introduced due to the MIS requirement. To precisely locate the surgical instruments in FOV, a surgical instrument tip segmentation dataset with 5471 images is built and a modified real-time instance segmentation network is trained based on the dataset. Then, a semi-global matching (SGM) algorithm is utilized with the binocular camera to estimate the depth in FOV. To meet the joint limit constraint and maintain the dexterity of the robotic system, a task-priority-based control method is adopted. The simulation and experiment results demonstrate the feasibility of the proposed framework for instrument tracking with the robotic laparoscope.

Beyond Trade-Off: An Optimized Binocular Stereo Vision Based Depth Estimation Algorithm for Designing Harvesting Robot in Orchards

Depth estimation is one of the bottleneck parts for harvesting robots to determine whether the operation of grasping or picking succeeds or not directly. This paper proposed a novel disparity completion method combined with bilateral filtering and pyramid fusion to improve the issues of incorrect outputs due to the missed or wrong matching when achieving 3D position from 2D images in open-world environments. Briefly, our proposed method has two significant advantages in general. Firstly, occlusion between leaves, branches, and fruits is a universal phenomenon in unstructured orchard environments, which results in the most depth estimation algorithms facing great challenges to obtain accurate outputs in these occluded regions. To alleviate these issues, unlike other research efforts that already exist, we optimized the semi-global matching algorithm to obtain high accuracy sparse values as an initial disparity map; then, an improved bilateral filtering algorithm is proposed to eliminate holes and discontinuous regions caused by occlusion to obtain precise and density disparity outputs. Secondly, due to taking the practical high-efficiency requirements of the automated harvesting robot in its working status into consideration, we attempted to merge multiple low-resolution bilateral filtering results through the pyramid fusion model which goes beyond the trade-off mechanism to improve the performance of both accuracy and time cost. Finally, a prototype harvesting robot was designed to conduct experiments at three kinds of different distances (0.6~0.75 m, 1~1.2 m, and 1.6~1.9 m). Experiment results showed that our proposed method achieved density disparity maps and eliminated holes and discontinuous defects in the disparity map effectively. The average absolute error of our proposed method is 3.2 mm, and the average relative error is 1.79%. In addition, the time cost is greatly reduced more than 90%. Comprehensive experimental results demonstrate that our proposed algorithm provides a potential possibility for designing harvesting.

Hardware-efficient algorithm and architecture design with memory and complexity reduction for semi-global matching

Among the stereo matching algorithms, the semi-global matching (SGM) is an efficient and high-accuracy method. However, its huge demand for memory access and high computational complexity makes it difficult to achieve a real-time and efficient processing on hardware. Based on the spatial redundancy found in the matching cost, we propose some effective techniques to reduce the requirement of on-chip and off-chip memory, while simultaneously greatly lower the computational complexity. Experimental results present that the proposed SGM algorithm reduces the computational complexity by 71%–74% and has almost the same quality of disparity map compared with the original 8-path SGM. The proposed 3-path fully-pipelined architecture is implemented on the Xilinx VCU-106 with a throughput of 1920 × 1080/54 fps. We also synthesize and layout it with TSMC 40 nm standard library, leading to an area of 8.1 mm2 with throughput of 1920 × 1080/192 fps. The million disparity estimation per second (MDE/s) of the proposed design reaches up to 50,960, which outperforms conventional ASIC implementations.

Real-time and accurate monocular 3D sensor using the reference plane calibration and an optimized SGM based on opencl acceleration

Speckle projection profilometry (SPP), as a promising structured light projection technique, can achieve global unambiguous 3D measurement by projecting a single random speckle pattern. In addition, the projected speckle pattern is usually etched into the microstructure of highly integrated Vertical-Cavity Surface-Emitting Laser (VCSEL), which makes the hardware system compact enough to be mounted on mobile devices such as robots. However, since the stereo matching algorithm used in SPP involves high computational overhead, it usually runs in real-time on specially customized hardware platforms such as ASIC/FPGA, rather than general-purpose mobile platforms. In this paper, we propose a real-time and accurate 3D measurement method using a monocular 3D sensor based on the infrared speckle projection. Similar to Kinect v1, our sensor mainly consists of an IR dot projector and one IR camera for projecting and capturing speckle images synchronously. Low-cost and high-quality speckle projection is achieved by customizing the projection pattern of VCSEL and using the beam copy function of Diffractive Optical Elements (DOE). Based on the 3D imaging principle of monocular 3D sensors, a reference plane calibration method is proposed to obtain a high-quality reference speckle image for improving the monocular matching accuracy. Then, benefited from the local memory mechanism and multiple operating synchronizations on the OpenCL environment, an optimized semi-global matching (SGM) algorithm using GPU is presented to achieve efficient and accurate depth reconstruction dynamically. Within the measurement range of 0.8m (length) ×0.5m (width) ×1m (depth), the proposed method can achieve real-time and single-shot 3D imaging with an accuracy of 1.277 mm at 75 FPS on GTX 1060 and 15 FPS on ARM Mail G52(mobile platform).

Semi-Global Stereo Matching Algorithm Based on Multi-Scale Information Fusion

Semi-global matching (SGM) has been widely used in binocular vision. In spite of its good efficiency, SGM still has difficulties in dealing with low-texture regions. In this paper, an SGM algorithm based on multi-scale information fusion (MSIF), named SGM-MSIF, is proposed by combining multi-path cost aggregation and cross-scale cost aggregation (CSCA). Firstly, the stereo pairs at different scales are obtained by Gaussian pyramid down-sampling. The initial matching cost volumes at different scales are computed by combining census transform and color information. Then, the multi-path cost aggregation in SGM is introduced into the cost aggregation at each scale and the aggregated cost volumes are fused by CSCA. Thirdly, the disparity map is optimized by internal left-right consistency check and median filter. Finally, experiments are conducted on Middlebury datasets to evaluate the proposed algorithm. Experimental results show that the average error matching rate (EMR) of the proposed SGM-MSIF algorithm reduced by 1.96% compared with SGM. Compared with classical cross-scale stereo matching algorithm, the average EMR of SGM-MSIF algorithm reduced by 0.92%, while the processing efficiency increased by 58.7%. In terms of overall performance, the proposed algorithm outperforms the classic SGM and CSCA algorithms. It can achieve high matching accuracy and high processing efficiency for binocular vision applications, especially for those with low-texture regions.

Semi-Global Matching Assisted Absolute Phase Unwrapping.

Measuring speed is a critical factor to reduce motion artifacts for dynamic scene capture. Phase-shifting methods have the advantage of providing high-accuracy and dense 3D point clouds, but the phase unwrapping process affects the measurement speed. This paper presents an absolute phase unwrapping method capable of using only three speckle-embedded phase-shifted patterns for high-speed three-dimensional (3D) shape measurement on a single-camera, single-projector structured light system. The proposed method obtains the wrapped phase of the object from the speckle-embedded three-step phase-shifted patterns. Next, it utilizes the Semi-Global Matching (SGM) algorithm to establish the coarse correspondence between the image of the object with the embedded speckle pattern and the pre-obtained image of a flat surface with the same embedded speckle pattern. Then, a computational framework uses the coarse correspondence information to determine the fringe order pixel by pixel. The experimental results demonstrated that the proposed method can achieve high-speed and high-quality 3D measurements of complex scenes.

Optimization of greenhouse tomato localization in overlapping areas

Tomato localization is the main difficulty of tomato picking robots vision system. To provide robots vision system with the accurate position of tomatoes, this paper collects images with a binocular camera, provides the principle of binocular ranging, and improves the census stereo matching algorithm. The improved algorithm betters the area matching process: only the areas containing tomatoes are matched, more constraints are applied on the area matching, and the Localization of tomatoes in overlapping areas is optimized. Compared with stereo processing by semiglobal matching and mutual information (SGBM) algorithm and pyramid stereo matching network (PSMnet), the improved algorithm achieved an extremely small disparity error. The absolute error maximized at 4 pixels. The matching time for a single image was 10 ms at the most. In this way, the matching time is improved significantly. Experimental results show that the improved census matching algorithm provided tomato picking robots vision system with more accurate localization information, and greatly improved the picking efficiency.

Disparity Estimation Networks for Aerial and High-Resolution Satellite Images: A Review

This study reviews the PSM and HSM deep learning architectures for disparity estimation from an input stereo pair and assesses their applicability for satellite stereo reconstruction. All methods are tested on urban landscapes unseen at training time, using pre-trained weights learned from a stereo matching benchmark for aerial imagery. The quality of the disparity maps output by each method is assessed based on the subsequent surface models, which are evaluated using a lidar reference model. The conducted experiments give insight into the robustness of each architecture (e.g. robustness to different input resolutions, color spaces or acquisition dates), as well as their generalizability across different cities. Lastly, the results obtained with the different networks are compared with those of a state-of-the-art variant of the semi-global matching algorithm, which is a well-known classic methodology for satellite dense stereo matching. **This is an MLBriefs article, the source code has not been reviewed!**<br> **The original source codes are available [[here|https://github.com/JiaRenChang/PSMNet]] and [[here|https://github.com/gengshan-y/high-res-stereo]] (last checked 2022/11/14).**<br> <span style="color:red">**BEST STUDENT PAPER MLBRIEFS 2022**</span>.

Digital elevation model workflow improvements for the MarsSI platform and resulting orthorectified mosaic of Oxia Planum, the landing site of the ExoMars 2022 rover

In this publication, we explore the options to improve the quality and speed of DEM generation procedures from stereo-photogrammetry pipeline of the MarsSI online service. We study available algorithms, options and workflows. Taking inspiration on earth-based work, we propose a new workflow relying on the semi-global matching algorithms and a two steps approach with the result of artifacts/noise reduction in the products, and improved co-registration of the outputs. We use this new workflow in the production of mosaic of ground models and orthorectified images. We illustrate the pipeline with the example of the othorectified HiRISE images mosaic that has been used in the framework of Exomars project to perform a join mapping of Oxia Planum, the landing site of Exomars rover. • Describes the new DEM workflow of the MarsSI platform. • Noise and artifact levels were improved. • Alignment to MOLA was improved, enough to consider automated CTX mosaics. • ESA ExoMars 2022 landing site covering DEM and orthorectified image datasets.

Scale-Invariant Localization of Electric Vehicle Charging Port via Semi-Global Matching of Binocular Images

Automatic charging for electric vehicles has broad development prospects for meeting the personalized service experience of users while overcoming the inherent safety hazards. An identification and positioning approach suitable for engineering applications is the key to promoting automatic charging. In this paper, a low-cost, high-precision method to identify and position charging ports based on SIFT and SGBM is proposed. The feature extraction approach based on SIFT is adopted to produce the difference of Gaussian (DOG) for scale space construction, and the feature matching algorithm with nearest-neighbor search, which is a kind of machine learning, is utilized to yield the map set of matching points. In addition, the disparity calculation is conducted with a semi-global matching algorithm to obtain high-precision positioning results for the charging port. In order to verify the feasibility of the method, a complete identification and positioning experiment of charging port was carried out based on OpenCV and MATLAB.

ASSESSMENT OF STEREO-EXTRACTED DSM FROM WORLDVIEW-3 OVER DIFFERENT LAND COVERS DEPENDING ON THE IMAGING GEOMETRY

Abstract. WorldView-3's 0.31m resolution in panchromatic mode, makes it one of the highest resolution commercial satellite in the world. This fact, together with its excellent stereo capabilities, make this satellite ideal for digital surface model (DSM) extraction working on very complex morphologies where a higher level of detail is required. In this communication we assess the quality (both completeness and vertical accuracy) of DSM extracted from WorldView-3 stereo pairs depending on the image geometry and sun position. Three different land covers (bare soil, urban areas and agricultural plastic greenhouses) have been tested in the Southeast of Spain (Almería). The well-known semiglobal matching (SGM) algorithm was used for all the extracted DSM. A clear relationship between DSM completeness and the WorldView-3 stereo pair imaging geometry measured as convergence angle was found. The completeness values decreased as convergence angles increased, especially in complex reliefs. In fact, convergence angles lower than 16 degrees is recommended when working on urban or greenhouse land covers. Moreover, sun light can cause glint effect in greenhouse areas. In this land cover, the attained results suggest to use stereo pairs taken when the sun presented a very low elevation. In Almería, the last happens in winter. The best results in all the tested land covers can be obtained by fusing DSM extracted from more than one stereo pair.

FINE-TUNING DEEP LEARNING MODELS FOR STEREO MATCHING USING RESULTS FROM SEMI-GLOBAL MATCHING

Abstract. Deep learning (DL) methods are widely investigated for stereo image matching tasks due to their reported high accuracies. However, their transferability/generalization capabilities are limited by the instances seen in the training data. With satellite images covering large-scale areas with variances in locations, content, land covers, and spatial patterns, we expect their performances to be impacted. Increasing the number and diversity of training data is always an option, but with the ground-truth disparity being limited in remote sensing due to its high cost, it is almost impossible to obtain the ground-truth for all locations. Knowing that classical stereo matching methods such as Census-based semi-global-matching (SGM) are widely adopted to process different types of stereo data, we therefore, propose a finetuning method that takes advantage of disparity maps derived from SGM on target stereo data. Our proposed method adopts a simple scheme that uses the energy map derived from the SGM algorithm to select high confidence disparity measurements, at the same utilizing the images to limit these selected disparity measurements on texture-rich regions. Our approach aims to investigate the possibility of improving the transferability of current DL methods to unseen target data without having their ground truth as a requirement. To perform a comprehensive study, we select 20 study-sites around the world to cover a variety of complexities and densities. We choose well-established DL methods like geometric and context network (GCNet), pyramid stereo matching network (PSMNet), and LEAStereo for evaluation. Our results indicate an improvement in the transferability of the DL methods across different regions visually and numerically.