Cost Aggregation Method Research Articles

A Proposal for Lodging Judgment of Rice Based on Binocular Camera

Rice lodging is a crucial problem in rice production. Lodging during growing and harvesting periods can decrease rice yields. Practical lodging judgment for rice can provide effective reference information for yield prediction and harvesting. This article proposes a binocular camera-based lodging judgment method for rice in real-time. As a first step, the binocular camera and Inertial Measurement Unit (IMU) were calibrated. Secondly, Census and Grayscale Level cost features are constructed for stereo matching of left and right images. The Cross-Matching Cost Aggregation method is improved to compute the aggregation space in the LAB color space. Then, the Winner-Takes-All algorithm is applied to determine the optimal disparity for each pixel. A disparity map is constructed, and Multi-Step Disparity Refinement is applied to the disparity map to generate the final one. Finally, coordinate transformation obtains 3D world coordinates corresponding to pixels. IMU calculates the real-time pose of the binocular camera. A pose transformation is applied to the 3D world coordinates of the rice to obtain its 3D world coordinates in the horizontal state of the camera (pitch and roll angles are equal to 0). Based on the distance between the rice and the camera level, thresholding was used to determine whether the region to be detected belonged to lodging rice. The disparity map effect of the proposed matching algorithm was tested on the Middlebury Benchmark v3 dataset. The results show that the proposed algorithm is superior to the widely used Semi-Global Block Matching (SGBM) stereo-matching algorithm. Field images of rice were analyzed for lodging judgments. After the threshold judgment, the lodging region results were accurate and could be used to judge rice lodging. By combining the algorithms with binocular cameras, the research results can provide practical technical support for yield estimation and intelligent control of rice harvesters.

Linear Time Non-Local Cost Aggregation on Complementary Spatial Tree Structures

Studies on many dense correspondence tasks in the field of computer vision attempt to find spatially smooth results. A typical way to solve these problems is by smoothing the matching costs using edge-preserving filters. However, local filters generate locally optimal results, in that they only take the costs over a small support window into account, and non-local filters based on a minimum spanning tree (MST) tend to overuse the piece-wise constant assumption. In this paper, we propose a linear time non-local cost aggregation method based on two complementary spatial tree structures. The geodesic distances in both the spatial and intensity spaces along the tree structures are used to evaluate the similarity of pixels, and the final aggregated cost is the sum of the outputs from these two trees. The filtering output of a pixel on each tree can be obtained by recursively aggregating the costs along eight sub-trees with linear time complexity. The only difference between the filtering procedures on these two spatial tree structures is the order of the filtering. Experimental results in optical flow estimation and stereo matching on the Middlebury and KITTI datasets demonstrate the effectiveness and efficiency of our method. It turns out that our method outperforms typical non-local filters based on the MST in cost aggregation. Moreover, a comparison of handcrafted features and deep features learned by convolutional neural networks (CNNs) in calculating the matching cost is also provided. The code will be available soon.

Exploring the path of sending information about visual contexts of industrial prints

Abstract Aiming at the influence of visual contextual information communication on industrial printmaking, this paper elaborates visual communication theory and establishes a visual information communication model after analyzing the concept, characteristics and components of visual communication, which has a role in visual contextual information communication. The four elements, three important links and the characteristics of visual communication in the process of visual communication are analyzed from the perspective of communication. Based on the above theories, a model of information communication in web interface is constructed. The computational process of the linear binocular cost aggregation algorithm is simplified by using the asymmetric cost aggregation method, and the more robust TAD matching cost function is used to compensate for the lack of accuracy caused by the simplification, and the computational process of the asymmetric linear cost aggregation algorithm is parallelized and mapped to the platform. Experimental data show that the asymmetric cost aggregation approach to linear binocular cost aggregation algorithm can obtain parallax maps in constant running time, and the quality of parallax maps is close to or better than some global matching algorithms which are at the forefront of research, and has a good prospect in the field of real-time binocular vision. The visual web pages designed by this method can better stimulate the user’s desire for acceptance, and the web pages have better visual communication effect and higher practical value, and can create better knowledge benefits in industrial printmaking, with an overall knowledge conversion rate of 85%.

Asymmetric cost aggregation network for efficient stereo matching

AbstractCost aggregation is crucial to the accuracy of stereo matching. A reasonable cost aggregation algorithm should aggregate costs within homogeneous regions where pixels have the same or similar disparities. Otherwise, the estimated disparity map is prone to the well‐known edge‐fattening issue and the problem of losing fine structures. However, current state‐of‐the‐art convolutional neural networks (CNNs) mainly do cost aggregation with square‐kernel convolutional layers that learn to adjust their kernel elements to make the actual receptive fields of the aggregated costs adapt to homogeneous regions with various shapes. This is a mechanism that easily leads to the above issues due to the translation equivalence and content‐insensitivity properties of CNNs. To tackle these problems, a novel densely connected asymmetric convolution block (Dense‐ACB) based on asymmetric convolutions is proposed to explicitly construct receptive fields with various shapes, which effectively alleviates the issues caused by mismatching shapes of receptive fields and homogeneous regions. The proposed Dense‐ACB brings new insight to CNN‐based stereo matching networks. Based on the proposed cost aggregation method, an efficient and effective stereo matching network is built, which not only achieves competitive overall accuracy compared with state‐of‐the‐art models but also effectively alleviates the edge‐fattening problem and preserves fine structures.

A novel cell structure‐based disparity estimation for unsupervised stereo matching

It is well known that preserving depth edges is an effective solution for achieving the accurate disparity map in stereo matching, but many state-of-the-art methods do not preserve depth edges well. In order to solve it efficiently, the cell structure containing irregular and regular shape regions is designed to preserve depth edges. Based on the well-designed cell structure, a novel disparity estimation method for stereo matching is proposed, in which a two-layer disparity optimization method is proposed to refine the disparity plane; it includes the front-parallel disparities computation and slanted-surfaces disparity plane refinement. In the framework of front-parallel disparities computation, a tree-based cost aggregation method is presented to make full use of the segmentation information of cells and then performing semi-global cost aggregation. In the framework of slanted-surfaces disparity plane refinement, a new probability model is proposed that employs Bayesian inference for refining disparities in textureless, weak texture and occluded regions. Experimental results show that higher accuracy could be achieved via the proposed method compared with some known state-of-the-art stereo methods on KITTI 2015 and Middlebury dataset, which are the standard benchmarks for testing the stereo matching methods. It can also be indicated that the proposed method can produce accurate disparity map and have good generalization performance.

Local Stereo Matching Using Adaptive Cross-Region-Based Guided Image Filtering with Orthogonal Weights

Adaptive cross-region-based guided image filtering (ACR-GIF) is a commonly used cost aggregation method. However, the weights of points in the adaptive cross-region (ACR) are generally not considered, which affects the accuracy of disparity results. In this study, we propose an improved cost aggregation method to address this issue. First, the orthogonal weight is proposed according to the structural feature of the ACR, and then the orthogonal weight of each point in the ACR is computed. Second, the matching cost volume is filtered using ACR-GIF with orthogonal weights (ACR-GIF-OW). In order to reduce the computing time of the proposed method, an efficient weighted aggregation computing method based on orthogonal weights is proposed. Additionally, by combining ACR-GIF-OW with our recently proposed matching cost computation method and disparity refinement method, a local stereo matching algorithm is proposed as well. The results of Middlebury evaluation platform show that, compared with ACR-GIF, the proposed cost aggregation method can significantly improve the disparity accuracy with less additional time overhead, and the performance of the proposed stereo matching algorithm outperforms other state-of-the-art local and nonlocal algorithms.

Hybrid cost aggregation for dense stereo matching

Matching cost initialization and aggregation are two major steps in the stereo matching framework. For dense stereo matching, a matching cost needs to be computed at each pixel for all disparities within the search range so that it can be used to evaluate pixel-to-pixel correspondence. Cost aggregation connects the matching cost with a certain neighbourhood to reduce mismatches by a supporting smoothness term. This paper presents a hybrid cost aggregation method to overcome mismatches caused by textureless surface, depth-discontinuity areas, inconsistent lightings in an image. The steps taken to aggregate costs for an energy function include adaptive support regions, multi-path aggregation, and adaptive penalties to generate a more accurate disparity map. Compared with two top-ranked stereo matching algorithms, the proposed algorithm yielded the disparity maps of the dataset in Middlebury benchmark V2 with smaller error ratios in depth-discontinuity regions.

Improved Cost Computation and Adaptive Shape Guided Filter for Local Stereo Matching of Low Texture Stereo Images

Dense stereo matching has been widely used in photogrammetry and computer vision applications. Even though it has a long research history, dense stereo matching is still challenging for occluded, textureless and discontinuous regions. This paper proposed an efficient and effective matching cost measurement and an adaptive shape guided filter-based matching cost aggregation method to improve the stereo matching performance for large textureless regions. At first, an efficient matching cost function combining enhanced image gradient-based matching cost and improved census transform-based matching cost is introduced. This proposed matching cost function is robust against radiometric variations and textureless regions. Following this, an adaptive shape cross-based window is constructed for each pixel and a modified guided filter based on this adaptive shape window is implemented for cost aggregation. The final disparity map is obtained after disparity selection and multiple steps disparity refinement. Experiments were conducted on the Middlebury benchmark dataset to evaluate the effectiveness of the proposed cost measurement and cost aggregation strategy. The experimental results demonstrated that the average matching error rate on Middlebury standard image pairs is 9.40%. Compared with the traditional guided filter-based stereo matching method, the proposed method achieved a better matching result in textureless regions.

Efficient binocular stereo correspondence matching with 1-D Max-Trees

Extraction of depth from images is of great importance for various computer vision applications. Methods based on convolutional neural networks are very accurate but have high computation requirements, which can be achieved with GPUs. However, GPUs are difficult to use on devices with low power requirements like robots and embedded systems. In this light, we propose a stereo matching method appropriate for applications in which limited computational and energy resources are available. The algorithm is based on a hierarchical representation of image pairs which is used to restrict disparity search range. We propose a cost function that takes into account region contextual information and a cost aggregation method that preserves disparity borders. We tested the proposed method on the Middlebury and KITTI benchmark data sets and on the TrimBot2020 synthetic data. We achieved accuracy and time efficiency results that show that the method is suitable to be deployed on embedded and robotics systems.

Multiple data cost-based stereo matching method to generate dense disparity maps from images under radiometric variations

Stereo matching algorithms are capable of providing dense 3D information of the environment through two images taken simultaneously from a pair of cameras placed parallel to each other. Obtaining an accurate disparity map from a stereo image pair is a challenging task and also computationally expensive. If we take into consideration the environmental effect, then the difficulty of the task increases drastically. The authors try to overcome this problem by combining multiple stereo cost functions in the form of a linear equation. To reduce the computation time, a segmentation based cost aggregation method is followed in an attempt to produce an accurate disparity map even in the presence of radiometric variations in the images. The performance of the proposed algorithm is observed while varying the parameter 'α' between the cost functions and the number of segments in the images. Different image pairs from the Middlebury stereo dataset were considered here.

Multiple data cost-based stereo matching method to generate dense disparity maps from images under radiometric variations

Stereo matching algorithms are capable of providing dense 3D information of the environment through two images taken simultaneously from a pair of cameras placed parallel to each other. Obtaining an accurate disparity map from a stereo image pair is a challenging task and also computationally expensive. If we take into consideration the environmental effect, then the difficulty of the task increases drastically. The authors try to overcome this problem by combining multiple stereo cost functions in the form of a linear equation. To reduce the computation time, a segmentation based cost aggregation method is followed in an attempt to produce an accurate disparity map even in the presence of radiometric variations in the images. The performance of the proposed algorithm is observed while varying the parameter ‘’ between the cost functions and the number of segments in the images. Different image pairs from the Middlebury stereo dataset were considered her

Ring Difference Filter for Fast and Noise Robust Depth from Focus.

Depth from focus (DfF) is a method of estimating the depth of a scene by using information acquired through changes in the focus of a camera. Within the DfF framework of, the focus measure (FM) forms the foundation which determines the accuracy of the output. With the results from the FM, the role of a DfF pipeline is to determine and recalculate unreliable measurements while enhancing those that are reliable. In this paper, we propose a new FM, which we call the "ring difference filter" (RDF), that can more accurately and robustly measure focus. FMs can usually be categorized as confident local methods or noise robust non-local methods. The RDF's unique ring-and-disk structure allows it to have the advantages of both local and non-local FMs. We then describe an efficient pipeline that utilizes the RDF's properties. Part of this pipeline is our proposed RDF-based cost aggregation method, which is able to robustly refine the initial results in the presence of image noise. Our method is able to reproduce results that are on par with or even better than those of state-of-the-art methods, while spending less time in computation.

Cost Aggregation for Stereo Matching Using Total Generalized Variation With Fusion Tensor

Stereo matching methods have achieved remarkable improvements by exploiting various attempts. However, most stereo matching algorithms still suffer from problems such as ambiguous region and inherent ambiguities. In particular, some problems affecting cost aggregation step have the greatest impact on depth results. To resolve the above-mentioned problems, we propose a new cost aggregation method using the modified total generalized variation with fusion tensor. First, two kinds of diffusion tensors are extracted from the guidance color image and the guidance depth map. They are incorporated into an energy functional to obtain the total generalized variation. After formulating the final energy functional, it is optimized via a primal-dual energy minimization method. The performance of the proposed method is experimentally verified by qualitatively and quantitatively comparing the results to those of other algorithms.

Cost aggregation benchmark for light field depth estimation

Light field depth estimation has become a mature research topic and there are numerous algorithms introduced by various research groups. However, comprehensive and fair benchmark is difficult to apply because there are large step variances of the introduced algorithms. It is essential to analyze each step in the light field depth estimation so that it could help design better and more robust algorithms. Thus, a thorough analysis of cost aggregation is conducted in this paper to analyze the performance of various cost aggregation methods on light field depth estimation. A study on the parameter setting for each cost aggregation method is performed. Then, each cost aggregation with its optimal parameters is evaluated individually. Instead of using the standard rank system, this paper utilizes the weighted rank system based on the score difference on each criterion. Experimental results confirm that the guided-filter based method outperforms other methods in most evaluation criteria.

Deep self-guided cost aggregation for stereo matching

In this paper, we present a deep self-guided cost aggregation method used to obtain an accurate disparity map from a pair of stereo images. Conventional cost aggregation methods typically perform joint image filtering on each cost volume slice. Thus, a guidance image is necessary for the conventional methods to work effectively. However, a guidance image might be unreliable due to several distortions, such as noise, blur, radiometric variation. Based on our observations, each cost volume slice can guide itself based on the internal features. However, finding a direct mapping function from the initial and filtered cost volume slice without any guidance image is difficult. To solve this problem, we use an advanced deep learning technique to perform self-guided cost aggregation. Because of the absence of ground truth cost volume, we offer the solution for the dataset generation. Our proposed deep learning network consists of two sub-networks: dynamic weight network and descending filtering network. We integrate the feature reconstruction loss and the pixelwise mean square loss function to preserve the edge property. Experimental results show that the proposed method achieves better results even though it does not employ a guidance image.

Deep Stereo Matching With Explicit Cost Aggregation Sub-Architecture

Deep neural networks have shown excellent performance for stereo matching. Many efforts focus on the feature extraction and similarity measurement of the matching cost computation step while less attention is paid on cost aggregation which is crucial for stereo matching. In this paper, we present a learning-based cost aggregation method for stereo matching by a novel sub-architecture in the end-to-end trainable pipeline. We reformulate the cost aggregation as a learning process of the generation and selection of cost aggregation proposals which indicate the possible cost aggregation results. The cost aggregation sub-architecture is realized by a two-stream network: one for the generation of cost aggregation proposals, the other for the selection of the proposals. The criterion for the selection is determined by the low-level structure information obtained from a light convolutional network. The two-stream network offers a global view guidance for the cost aggregation to rectify the mismatching value stemming from the limited view of the matching cost computation. The comprehensive experiments on challenge datasets such as KITTI and Scene Flow show that our method outperforms the state-of-the-art methods.

Efficient cost aggregation for feature-vector-based wide-baseline stereo matching

■■■In stereo matching applications, local cost aggregation techniques are usually preferred over global methods due to their speed and ease of implementation. Local methods make implicit smoothness assumptions by aggregating costs within a finite window; however, cost aggregation is a time-consuming process. Furthermore, most existing local methods are based on pixel intensity values, and hence are not efficient with feature vectors used in wide-baseline stereo matching. In this paper, a new cost aggregation method is proposed, where a Per-Column Cost matrix is combined with a feature-vector-based weighting strategy to achieve both matching accuracy and computational efficiency. Here, the proposed cost aggregation method is applied with the DAISY feature descriptor for wide-baseline stereo matching; however, this method can also be applied to a fast growing number of stereo matching techniques that are based on feature descriptors. A performance comparison with several benchmark local cost aggregation approaches is presented, along with a thorough analysis of the time and storage complexity of the proposed method.

Image Super-Resolution Reconstruction Based on Disparity Map and CNN

This paper presents a new method for registration of digital micro-mirror device (DMD) cameras through image reconstruction. The method called dual-channel super-resolution reconstruction (DCSR) uses a dual-channel input convolutional neural network method to iteratively obtain high resolution images. The originality of this study relies on the input color weight disparity map that enables extracting edges cues and feature cues effectively during the super-resolution. The iterative initial disparity map is obtained by a cost aggregation method based on the color weights, and the model parameters are updated during the iterative process. To validate performance, a series of the comparison experiments is developed on different network structures and parameter settings for the diverse deconv-pooling layers, convolution kernel sizes, and activation function. Experimental results show that the DCSR method can solve the sub-pixel registration problem of DMD cameras and achieve a good image reconstruction effect. The DCSR performs higher accuracy and an efficient process compared with traditional reconstruction made by single-channel input convolution neural network. The spatial resolution of the reconstructed image is significantly improved, and the detail resolution is enhanced.

A local stereo matching algorithm based on weighted guided image filtering for improving the generation of depth range images

Stereo matching is a challenging problem and highly accurate depth image is important in different applications. The main problem is to estimate the correspondence between two pixels in a stereo pair. To solve this problem, in the last decade, several cost aggregation methods aimed at improving the quality of stereo matching algorithms have been introduced. We propose a new cost aggregation method based on weighted guided image filtering (WGIF) for local stereo matching. The proposed algorithm solves multi-label problems in three steps. First, the cost volume is constructed using pixel-wise matching cost computation functions. Then, each slice of the cost volume is independently filtered using the WGIF, which substitutes for the smoothness term in the energy function. Finally, the disparity of any pixel is simply computed. The WGIF uses local weights based on a variance window of pixels in a guidance image for cost volume filtering. Experimental results using Middlebury stereo benchmark verify that the proposed method is effective due to a high quality cost volume filter.

3D cost aggregation with multiple minimum spanning trees for stereo matching.

Cost aggregation is one of the key steps in the stereo matching problem. In order to improve aggregation accuracy, we propose a cost-aggregation method that can embed minimum spanning tree (MST)-based support region filtering into PatchMatch 3D label search rather than aggregating on fixed size patches. However, directly combining PatchMatch label search and MST filtering is not straightforward, due to the extremely high complexity. Thus, we develop multiple MST structures for cost aggregation on plenty of 3D labels, and design the tree-level random search strategy to find possible 3D labels of each pixel. Extensive experiments show that our method reaches higher accuracy than the other existing cost-aggregation and global-optimization methods such as the 1D MST, the PatchMatch and the PatchMatch Filter, and currently ranks first on the Middlebury 3.0 benchmark.