Middlebury Dataset Research Articles

Structure tensor-based Gaussian kernel edge-adaptive depth map refinement with triangular point view in images

<span lang="EN-US">Image reconstruction is the process of restoring the image resolution. In 3D image reconstruction, the objects in different viewpoints are processed with the triangular point view (TPV) method to estimate object geometry structure for 3D model. This work proposes a depth refinement methodology in preserving the geometric structure of objects using the structure tensor method with a Gaussian filter by transforming a series of 2D input images into a 3D model. The computation of depth map errors can be found by comparing the masked area/patch with the distribution of the original image's greyscale levels using the error pixel-based patch extraction algorithm. The presence of errors in the depth estimation could seriously deteriorate the quality of the 3D effect. The depth maps were iteratively refined based on histogram bins number to improve the accuracy of initial depth maps reconstructed from rigid objects. The existing datasets such as the dataset tanks and unit (DTU) and Middlebury datasets, were used to build the model out of the object scene structure. The results of this work have demonstrated that the proposed patch analysis outperformed the existing state of the art models depth refinement methods in terms of accuracy.</span>

Advances in Real-Time 3D Reconstruction for Medical Endoscopy.

This contribution is intended to provide researchers with a comprehensive overview of the current state-of-the-art concerning real-time 3D reconstruction methods suitable for medical endoscopy. Over the past decade, there have been various technological advancements in computational power and an increased research effort in many computer vision fields such as autonomous driving, robotics, and unmanned aerial vehicles. Some of these advancements can also be adapted to the field of medical endoscopy while coping with challenges such as featureless surfaces, varying lighting conditions, and deformable structures. To provide a comprehensive overview, a logical division of monocular, binocular, trinocular, and multiocular methods is performed and also active and passive methods are distinguished. Within these categories, we consider both flexible and non-flexible endoscopes to cover the state-of-the-art as fully as possible. The relevant error metrics to compare the publications presented here are discussed, and the choice of when to choose a GPU rather than an FPGA for camera-based 3D reconstruction is debated. We elaborate on the good practice of using datasets and provide a direct comparison of the presented work. It is important to note that in addition to medical publications, publications evaluated on the KITTI and Middlebury datasets are also considered to include related methods that may be suited for medical 3D reconstruction.

Stereo matching from monocular images using feature consistency

AbstractSynthetic images facilitate stereo matching. However, synthetic images may suffer from image distortion, domain bias, and stereo mismatch, which would significantly restrict the widespread use of stereo matching models in the real world. The first goal in this paper is to synthesize real‐looking images for minimizing the domain bias between the synthesized and real images. For this purpose, sharpened disparity maps are produced from a mono real image. Then, stereo image pairs are synthesized using these imperfect disparity maps and the single real image in the proposed pipeline. Although the synthesized images are as realistic as possible, the domain styles of the synthesized images are always very different from the real images. Thus, the second goal is to enhance the domain generalization ability of the stereo matching network. For that, the feature extraction layer is replaced with a teacher–student model. Then, a constraint of binocular contrast features is imposed on the output of the model. When tested on the KITTI, ETH3D, and Middlebury datasets, the accuracy of the method outperforms traditional methods by at least 30%. Experiments demonstrate that the approaches are general and can be conveniently embedded into existing stereo networks.

Binocular Visual Measurement Method Based on Feature Matching.

To address the issues of low measurement accuracy and unstable results when using binocular cameras to detect objects with sparse surface textures, weak surface textures, occluded surfaces, low-contrast surfaces, and surfaces with intense lighting variations, a three-dimensional measurement method based on an improved feature matching algorithm is proposed. Initially, features are extracted from the left and right images obtained by the binocular camera. The extracted feature points serve as seed points, and a one-dimensional search space is established accurately based on the disparity continuity and epipolar constraints. The optimal search range and seed point quantity are obtained using the particle swarm optimization algorithm. The zero-mean normalized cross-correlation coefficient is employed as a similarity measure function for region growing. Subsequently, the left and right images are matched based on the grayscale information of the feature regions, and seed point matching is performed within each matching region. Finally, the obtained matching pairs are used to calculate the three-dimensional information of the target object using the triangulation formula. The proposed algorithm significantly enhances matching accuracy while reducing algorithm complexity. Experimental results on the Middlebury dataset show an average relative error of 0.75% and an average measurement time of 0.82 s. The error matching rate of the proposed image matching algorithm is 2.02%, and the PSNR is 34 dB. The algorithm improves the measurement accuracy for objects with sparse or weak textures, demonstrating robustness against brightness variations and noise interference.

Dark channel prior in low frequency domain for time-efficient single image dehazing

ABSTRACT Dehazing plays a vital role in enhancing the quality of outdoor images for computer-based applications. While dehazing algorithms significantly improve outdoor image quality for computer vision tasks, their effectiveness hinges on two crucial factors: 1. Speed of dehazing operations 2. Platform-agnostic nature of the chosen algorithm. This study introduces a novel Dark Channel Prior-Low Frequency Domain (DCP-LFD) method designed to effectively and efficiently eliminate haze from single hazy images. The method employs a hybrid strategy, combining DCP for haze reduction in the low-frequency domain and Soft-thresholding for enhancing high-frequency coefficients. This dual-pronged approach enables improvement of image quality in low and high-frequency domains, contributing to better dehazing outcomes taking less computation time. The proposed method’s effectiveness is accessed using metrics like PSNR and SSIM, comparing its performance against state-of-the-art dehazing methods across five benchmark datasets: SOTS, RTTS and HSTS subsets from RESIDE, HazeRD, and Middlebury datasets. No-Reference Image Quality Assessments viz. BRISQUE and FADE are performed on RTTS and private dataset to objectively evaluate dehazed image quality. A comparison of dehazing time is executed, highlighting the method’s proficiency in achieving faster results. This analysis accentuates the method’s capacity to efficiently remove haze from outdoor images while emphasising its time-saving efficiency.

Multi-Scale Interaction Network for Low-Light Stereo Image Enhancement

Low-light stereo image enhancement (LLSIE) is a task that improves the visual quality of stereo images captured in low-light environment. The core idea of LLSIE is to exploit the cross-view cues for enhancing the stereo image pairs. However, LLSIE is still a challenging task due to the perspective difference and insufficient information interaction between left and right views. In this paper, we propose a novel Multi-Scale Interaction Network (MSINet) for LLSIE. To be specific, we present a new multi-scale stereo Interaction module (MIM), which can promote the multi-scale cross-view information flow. Firstly, MIM extracts multi-scale features from both the left and right views. Then, the features from two views are fused in different scales, so that the information from stereo image pairs can be fully exchanged. Compared to the existing parallax attention module (PAM), our MIM explores the cross-view interaction in multiple scales with less computational cost more sufficiently. Besides, we also develop a new Spatial Gating Block (SGB) for refining intra-view features. By incorporating the long residual and spatial gating mechanism, SGB is capable of completing dynamic spatial modeling and obtaining strong feature representations. Extensive experiments on the Flickr1024, KITTI 2015 and Middlebury datasets show that the proposed MSINet reaches SOTA performance and obtains better visual results than other compared methods.

Stereo matching algorithm based on improved census transform and minimum spanning tree cost aggregation

Existing stereo matching algorithms suffer from issues such as susceptibility to distortion, weak noise resistance, and a high rate of mismatches in regions with weak textures and discontinuous disparities. To address these challenges, this paper proposes a stereo matching algorithm based on an improved census transform and minimum spanning tree (MST) cost aggregation. In the cost calculation phase, we employ a Gaussian-weighted transformation window and incorporate gradient and edge information to perform weighted fusion of the results. In the cost aggregation process, we introduce a collaborative adaptive window. Each pixel acquires information from the support window of the guided filter (GF) and other pixels within the MST. Furthermore, we integrate the SLIC superpixel segmentation algorithm into MST construction. These two components work synergistically to assign appropriate adaptive weights to pixels, facilitating coordinated cost volume aggregation. Different optimization methods are applied to address mismatched points of various types in post-disparity processing.Performance evaluation using the Middlebury dataset and KITTI dataset demonstrates that our proposed algorithm not only enhances matching accuracy in regions with discontinuous disparities and weak textures but also exhibits significantly improved robustness to interference. Additionally, the resulting disparity map displays smoother edges.

An application of stereo matching algorithm based on transfer learning on robots in multiple scenes

Robot vision technology based on binocular vision holds tremendous potential for development in various fields, including 3D scene reconstruction, target detection, and autonomous driving. However, current binocular vision methods used in robotics engineering have limitations such as high costs, complex algorithms, and low reliability of the generated disparity map in different scenes. To overcome these challenges, a cross-domain stereo matching algorithm for binocular vision based on transfer learning was proposed in this paper, named Cross-Domain Adaptation and Transfer Learning Network (Ct-Net), which has shown valuable results in multiple robot scenes. First, this paper introduces a General Feature Extractor to extract rich general feature information for domain adaptive stereo matching tasks. Then, a feature adapter is used to adapt the general features to the stereo matching network. Furthermore, a Domain Adaptive Cost Optimization Module is designed to optimize the matching cost. A disparity score prediction module was also embedded to adaptively adjust the search range of disparity and optimize the cost distribution. The overall framework was trained using a phased strategy, and ablation experiments were conducted to verify the effectiveness of the training strategy. Compared with the prototype PSMNet, on KITTI 2015 benchmark, the 3PE-fg of Ct-Net in all regions and non-occluded regions decreased by 19.3 and 21.1% respectively, meanwhile, on the Middlebury dataset, the proposed algorithm improves the sample error rate at least 28.4%, which is the Staircase sample. The quantitative and qualitative results obtained from Middlebury, Apollo, and other datasets demonstrate that Ct-Net significantly improves the cross-domain performance of stereo matching. Stereo matching experiments in real-world scenes have shown that it can effectively address visual tasks in multiple scenes.

Bidirectional Optical Flow NeRF: High Accuracy and High Quality under Fewer Views

Neural Radiance Fields (NeRF) can implicitly represent 3D-consistent RGB images and geometric by optimizing an underlying continuous volumetric scene function using a sparse set of input views, which has greatly benefited view synthesis tasks. However, NeRF fails to estimate correct geometry when given fewer views, resulting in failure to synthesize novel views. Existing works rely on introducing depth images or adding depth estimation networks to resolve the problem of poor synthetic view in NeRF with fewer views. However, due to the lack of spatial consistency of the single-depth image and the poor performance of depth estimation with fewer views, the existing methods still have challenges in addressing this problem. So this paper proposes Bidirectional Optical Flow NeRF(BOF-NeRF), which addresses this problem by mining optical flow information between 2D images. Our key insight is that utilizing 2D optical flow images to design a loss can effectively guide NeRF to learn the correct geometry and synthesize the right novel view. We also propose a view-enhanced fusion method based on geometry and color consistency to solve the problem of novel view details loss in NeRF. We conduct extensive experiments on the NeRF-LLFF and DTU MVS benchmarks for novel view synthesis tasks with fewer images in different complex real scenes. We further demonstrate the robustness of BOF-NeRF under different baseline distances on the Middlebury dataset. In all cases, BOF-NeRF outperforms current state-of-the-art baselines for novel view synthesis and scene geometry estimation.

Method of depth simulation imaging and depth image super-resolution reconstruction for a 2D/3D compatible CMOS image sensor.

This paper presents a depth simulation imaging and depth image super-resolution (SR) method for two-dimensional/three-dimensional compatible CMOS image sensors. A depth perception model is established to analyze the effects of depth imaging parameters and evaluate the real imaging effects. We verify its validity by analyzing the depth error, imaging simulation, and auxiliary physical verification. By means of the depth simulation images, we then propose a depth SR reconstruction algorithm to recover the low-resolution depth maps to the high-resolution depth maps in two types of datasets. With the best situation in depth accuracy kept, the root mean square error (RMSE) of Middlebury dataset images are 0.0156, 0.0179, and 0.0183m. The RMSE of RGB-D dataset images are 0.0223 and 0.0229m. Compared with other listed conventional algorithms, our algorithm reduces the RMSE by more than 16.35%, 17.19%, and 23.90% in the Middlebury dataset images. Besides, our algorithm reduces the RMSE by more than 9.71% and 8.76% in the RGB-D dataset images. The recovery effects achieve optimized results.

Accurate edge-preserving stereo matching by enhancing anisotropy

Stereo matching has attracted substantial research attention, as it plays an important role in many visual tasks such as autonomous driving. Despite the remarkable progress has been made by all kinds of stereo matching methods, they still suffer in some challenging scenes like disparity discontinuities. The root cause of blurred depth boundaries lies in information diffusion across edge boundaries. Aiming at this problem, this paper presents a novel edge-preserving stereo matching method by enforcing anisotropy, where information is allowed to propagate throughout the image except through large discontinuities. Specifically, we propose an intra-scale cost aggregation algorithm to smooth out noise in homogeneous regions while well preserving strong edges in the guided image. And weighted averaging scheme, where weights are calculated according to the variances of pixels in different overlapping support windows, is utilized for enhancing anisotropy to improve accuracy at disparity discontinuities. We carry out comprehensive experiments on Middlebury public datasets to demonstrate the accuracy and edge-preserving property of our method. Qualitative and quantitative performance evaluation on Middlebury data sets demonstrate the superior performance of our method for strong edge preservation with 20.77% decreases in wrong matching ratios in discontinuous regions.

Performance evaluation of the non-local cost aggregation in stereo matching algorithm

This paper presents the evaluation of the efficiency and accuracy of the stereo matching algorithm using a census transform with the non-local aggregation technique. This technique is proposed to reduce the average error of the overall matching process in order to produce an accurate disparity map and to access the efficiency of the matching algorithm. Fundamentally, the stereo matching process highly depends on the quality of the image pair. The properties different between left and right images such as image brightness, exposure, occlusion and less texture area have a significant influence on the matching algorithm's accuracy. Thus, this paper analysed the performance of the stereo matching algorithm using Census Transform at matching cost computation, the non-local technique of cost aggregation and winner-take-all strategy at cost optimization. The initial disparity value obtained in cost optimization is used as the final disparity map. The disparity maps obtained from three different aggregation methods are evaluated in terms of accuracy and processing time. The experimental results using the Middlebury dataset show that the non-local method is more accurate compared with other aggregation methods and while maintaining the efficiency of the algorithm.

Semi-Global Stereo Matching Algorithm Based on Multi-Scale Information Fusion

Semi-global matching (SGM) has been widely used in binocular vision. In spite of its good efficiency, SGM still has difficulties in dealing with low-texture regions. In this paper, an SGM algorithm based on multi-scale information fusion (MSIF), named SGM-MSIF, is proposed by combining multi-path cost aggregation and cross-scale cost aggregation (CSCA). Firstly, the stereo pairs at different scales are obtained by Gaussian pyramid down-sampling. The initial matching cost volumes at different scales are computed by combining census transform and color information. Then, the multi-path cost aggregation in SGM is introduced into the cost aggregation at each scale and the aggregated cost volumes are fused by CSCA. Thirdly, the disparity map is optimized by internal left-right consistency check and median filter. Finally, experiments are conducted on Middlebury datasets to evaluate the proposed algorithm. Experimental results show that the average error matching rate (EMR) of the proposed SGM-MSIF algorithm reduced by 1.96% compared with SGM. Compared with classical cross-scale stereo matching algorithm, the average EMR of SGM-MSIF algorithm reduced by 0.92%, while the processing efficiency increased by 58.7%. In terms of overall performance, the proposed algorithm outperforms the classic SGM and CSCA algorithms. It can achieve high matching accuracy and high processing efficiency for binocular vision applications, especially for those with low-texture regions.

Super-resolution reconstruction of binocular image based on multi-level fusion attention network

Super-resolution reconstruction of binocular image based on multi-level fusion attention network

StereoPairFree: Self-Constructed Stereo Correspondence Network From Natural Images

This article proposes a variant of a convolutional neural network-based stereo matching method, StereoPairFree, that requires only normal and not left or right stereo images. The corresponding image patches for training stereo matching cost networks look similar, apart from some properties, such as different illumination, occlusion, distortion, and foreshortening. We propose a method for generating synthesized pairs of corresponding image patches for a given image patch. We also propose a multipatch matching cost network that exploits various input patch sizes. The proposed matching cost network is optimized by cross-based cost aggregation and semiglobal matching, followed by consistency checks and bilateral filtering. Hence, StereoPairFree does not require a single stereo pair but standard images to build a deep learning stereo matching method. It is the first stereo matching method constructed from normal images and significantly outperforms monocular stereo matching approaches (Hur and Roth, 2020), (Hur and Roth, 2021), (Schuster et al., 2020). We evaluated StereoPairFree with the KITTI-2012, KITTI-2015, and Middlebury datasets. It significantly outperformed the baseline and several deep learning-based methods using these datasets.

Dense Scene Flow Estimation with the GVF Snake Model for Obstacle Detection Using a 3D Sensor in the Path-Planning Module

Abstract The novel framework for estimating dense scene flow using depth camera data is demonstrated in the article. Using these estimated flow vectors to identify obstacles improves the path planning module of the autonomous vehicle's (AV) intelligence. The primary difficulty in the development of AVs has been thought to be path planning in cluttered environments. These vehicles must possess the intelligence to recognize their surroundings and successfully navigate around obstacles. The AV needs a thorough understanding of the surroundings to detect and avoid obstacles in a cluttered environment. Therefore, when determining the course, it is preferable to be aware of the kinematic behavior (position and the direction) of the obstacles. As a result, by comparing the depth images between different time frames, the position and direction of the obstacles are calculated using a 3D vision sensor. The current study focuses on the extraction of the flow vectors in 3D coordinates from the differential scene flow method. Generally, the evaluation of scene flow algorithms is crucial in determining their accuracy and effectiveness in different applications. The gradient of the vector field snake model, which extracts changes in pixel values in three directions, is combined with the scene flow technique to identify both static and dynamic obstacles. Our goal is to create a single-vision sensor-based real-time obstacle avoidance method based on scene flow estimation. In addition, the common evaluation metrics such as endpoint error (EPE), average angular error (AAE), and standard deviation angular error (STDAE) are used to measure the accuracy of different algorithms in terms of computational errors with the benchmark Middlebury datasets. The proposed technique is validated with different experiments using a Pixel-Mixed-Device (PMD) camera and a Kinect sensor as 3D sensors. Finally, the numerical and experimental results are displayed and reported.

An efficient real-time accelerator for high-accuracy DNN-based optical flow estimation in FPGA

Recently, accelerator architectures of deep neural networks (DNNs) have been designed to accelerate computer vision tasks, gaining the advantages of both accuracy and speed. However, DNNs-based optical flow estimation accelerators have not been successfully deployed due to the resource constraint. Existing hardware accelerators for optical flow estimation are all designed for non-DNN methods and generally perform poorly in estimated accuracy. In this paper, we present FlowAcc, a dedicated hardware accelerator for DNN-based optical flow estimation, adopting a pipelined hardware design for the real-time processing of image streams. We design an efficient multiplexing binary neural network (BNN) architecture for pyramidal feature extraction to significantly reduce the hardware cost and make it independent of the pyramid-level number. Furthermore, efficient designs such as hamming distance calculation and competent flow regularization are used for hierarchical optical flow estimation to greatly improve the system’s efficiency. Comprehensive experimental results demonstrate that FlowAcc achieves state-of-the-art estimation accuracy and real-time performance on the Middlebury dataset when compared with the existing optical flow accelerators.

ClarifyNet: A high-pass and low-pass filtering based CNN for single image dehazing

Dehazing refers to removing the haze and restoring the details from hazy images. In this paper, we propose ClarifyNet, a novel, end-to-end trainable, convolutional neural network architecture for single image dehazing. We note that a high-pass filter detects sharp edges, texture, and other fine details in the image, whereas a low-pass filter detects color and contrast information. Based on this observation, our key idea is to train ClarifyNet on ground-truth haze-free images, low-pass filtered images, and high-pass filtered images. Based on this observation, we present a shared-encoder multi-decoder model ClarifyNet which employs interconnected parallelization. While training, ground-truth haze-free images, low-pass filtered images, and high-pass filtered images undergo multi-stage filter fusion and attention. By utilizing a weighted loss function composed of SSIM loss and L1 loss, we extract and propagate complementary features. We comprehensively evaluate ClarifyNet on I-HAZE, O-HAZE, Dense-Haze, NH-HAZE, SOTS-Indoor, SOTS-Outdoor, HSTS, and Middlebury datasets. We use PSNR and SSIM metrics and compare the results with previous works. For most datasets, ClarifyNet provides the highest scores. On using EfficientNet-B6 as the backbone, ClarifyNet has 18 M parameters (model size of ∼71 MB) and a throughput of 8 frames-per-second while processing images of size 2048 × 1024.

Stereo matching by using a new local descriptor based on sign and order in belief propagation algorithm

Stereo matching is still a challenging problem in computer vision. Belief propagation (BP) algorithm is one of the global methods to solve this problem which can lead to desired results if useful information is applied in data function of this algorithm. In most literatures, color information of pixels has been applied in data function of BP which its result is not very well. In this paper, a low complexity modified version of local descriptor based on contrast and ordering (LCO) of intensity differences is proposed which is known as the local sign and ordering (LSO). LSO, like its original descriptor, has some distinguished properties such as robustness against illumination change and preserving depth discontinuity which are the important challenges of stereo matching. Moreover, a combination of intensity values of pixels and LSO descriptor is presented. This combination of cost measures is utilized as data function of BP algorithm. In the post-processing of stereo matching, the left–right consistency check is applied to the final result, which is followed by median filter in order to refine disparity map. Performed experiments on the Middlebury dataset show that the results of the proposed method are competitive with the methods based on other descriptors used in BP algorithm. Furthermore, the proposed method has relatively good performance compared to the other methods rather than BP algorithm.

Accuracy analysis of lossless and lossy disparity map compression

In this paper, we analysed lossless and lossy compression of disparity (depth) images with low range resolution. For that goal, the well-known publicly available Middlebury dataset is used with stereo image pairs, their disparity ground truths and disparity estimations obtained using state-of-the-art algorithms. We show that the WebP image format is suitable for lossless compression of disparity images, with compression ratios between 14 and 56 and a mean compression ratio of 20. Much higher compression ratios, better than 60, can be achieved using lossy image compression HEIC algorithm, with acceptable reduction of the disparity map accuracy. This high compression ratio is proportional to the transmission time reduction.