Cost Aggregation Research Articles

SPNet: Learning Stereo Matching With Slanted Plane Aggregation

Matching cost aggregation plays a critical role for the stereo matching task. Existing CNN-based methods commonly use 3D convolutions to aggregate matching costs from a local 3D space. However, the high computational cost of 3D convolutions limits their applications. Traditional methods show that a slanted support window in 3D space can help to aggregate matching costs from informative regions, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.,</i> the surfaces of objects. Motivated by this idea, we propose a SPNet with differentiable slanted plane aggregation. Our slanted plane aggregation layers aggregate matching costs from a learnable slanted plane in a local 3D space to reduce computational and memory costs. Experimental results show that our slanted plane aggregation layers can learn to fit the surfaces of objects and effectively aggregate matching costs. Comparison with previous stereo matching methods shows that our network achieves competitive performance with higher efficiency.

ESG and Sustainable Agriculture : Focus on 'True Cost' of Agricultural Operations

In every economy, agriculture is a central activity which has significant impact on environmental, social and economic aspects of the country. This article presents and highlights the ‘true cost of agricultural operation’, a system of compilation and aggregation of all relevant costs, including externalities, which not only limit itself to the financial metrics but go beyond the broader considerations of ESG factors and sustainability. To arrive at the true cost of agricultural operations, global concerns about ESG and sustainability aspects necessitate a transition from “incremental change” to “transformational change.”

Development of Methods for Route Optimization of Work in Inhomogeneous Radiation Fields to Minimize the Dose Load of Personnel

The importance of the optimization principle implementation in ensuring the radiation protection of NPP personnel was emphasized. The potential of route optimization in reducing the dose load of the personnel of nuclear power plants and other nuclear facilities is shown. The paper considers the main directions of the authors’ development of the theory and algorithms of route optimization of work in inhomogeneous radiation fields during maintenance, repair, modernization, dismantling of NPP equipment, and elimination of the radiation accident consequences. The results of the computational experiments that were carried out with the “Uran” supercomputer of the IMM UB RAS for the checking of the developed algorithms are presented. The article provides an overview of the developed methods of route optimization of work using the dynamic programming method, including consideration of the constraints in the form of precedence conditions, which means the requirement to perform certain tasks only after the completion of others. Dijkstra’s method was used to solve the “dosimetrist’s problem”, where the optimal route for the dosimetrist’s movement is being constructed, including obstacles bypassing and visiting specified points in the room where it is necessary to perform work to determine the radiation environment characteristics such as measuring the radiation dose rate, taking samples, etc. The routing of movements with the non-additive aggregation of costs is considered. The content of the problem is shown on the example of the radiation accident consequences eliminating on a locality, where, as a result of radioactive fragments scattering, a system of emitting elements appears, which must be deactivated, i.e., dismantled or screened. This task must be carried out in consecutive cycles with a definite threshold level of personnel exposure per shift. A characteristic feature is the dependence of cost functions (here, dosimetric cost) on the list of tasks: only sources that have not been dismantled yet continue emitting at the moment. Precedence conditions are also possible.

Adaptive Aggregate Stereo Matching Network with Depth Map Super-Resolution.

In order to avoid the direct depth reconstruction of the original image pair and improve the accuracy of the results, we proposed a coarse-to-fine stereo matching network combining multi-level residual optimization and depth map super-resolution (ASR-Net). First, we used the u-net feature extractor to obtain the multi-scale feature pair. Second, we reconstructed global disparity in the lowest resolution. Then, we regressed the residual disparity using the higher-resolution feature pair. Finally, the lowest-resolution depth map was refined by using the disparity residual. In addition, we introduced deformable convolution and group-wise cost volume into the network to achieve adaptive cost aggregation. Further, the network uses ABPN instead of the traditional interpolation method. The network was evaluated on three datasets: scene flow, kitti2015, and kitti2012 and the experimental results showed that the speed and accuracy of our method were excellent. On the kitti2015 dataset, the three-pixel error converged to 2.86%, and the speed was about six times and two times that of GC-net and GWC-net.

Symmetry Weight-sharing for Patch-based Stereo Matching

Neural networks are becoming more popular than traditional methods in stereo matching. The networks can be decomposed into four sub-modules: feature extraction / matching cost computation, cost aggregation, disparity computation / optimization, and disparity refinement. A typical design for the feature extraction networks is that the left and right branches share the same weights. However, the Siamese networks are weak at distinguishing neighboring patches because of the interference of geometric distortion on slanted surfaces. This paper proposes symmetry weight-sharing to improve the feature extraction networks. The geometry of feature extraction and patch comparison has been analyzed, which shows that symmetry weight-sharing can fulfill the geometry on slanted surfaces. A half-translation module is proposed to implement symmetry weight-sharing without additional computational costs. Experiments on the KITTI 2012 and KITTI 2015 datasets show that the symmetry weight-sharing networks have better performance than the weight-sharing networks.

Stereo matching algorithm based on combined matching cost computation and edge preserving filters

The stereo matching &lt;span&gt;process is one of the key areas that impact the stereo vision technologies which are commonly used in the application of three-dimensional reconstructions. The accuracy of the depth information used in three-dimensional reconstruction is directly proportional to the accuracy of the disparity obtained from stereo matching. The challenging issue in the stereo matching process is to determine the accurate corresponding point between the left image and right image, especially for image pairs that have different exposure such as different illumination and image pair with less texture region. In order to increase the accuracy of disparity value, a new stereo matching algorithm is proposed based on the combination of Sum of absolute different and census transform at matching cost computation. guided filter was used in the matching cost aggregation in order to remove noise and preserve the edge of the image. In the optimization step, the winner take all strategy is used to select the minimum matching cost. Finally, a median filter is applied to the initial disparity map for refinement purposes. The experimental results show that the algorithm is effective in reducing the error and improving the accuracy of the disparity map in different illumination regions, less textured regions and different environmental exposure.&lt;/span&gt;

Binocular Images Dense Matching considering Image Adaptive Color Weights and Feature Points

When the matching cost function in Semiglobal Matching is unstable, the inaccurate matching cost values will be propagated in the cost aggregation process. It will lead to a serious mismatching phenomenon. To address the problem, a binocular images dense matching method considering image adaptive color weights and feature points was proposed. Firstly, The Color Birchfield Tomasi (CBT) matching cost calculation method was proposed to obtain a stable initial cost volume, which combined image adaptive color weights and gradient information. Secondly, the Scale-invariant Feature Transform matching algorithm was used to extract the a priori feature points from binocular images. Then, the feature points were filtrated. The cost volume was optimized by using their coordinate information and disparity information. Finally, an aggregation path segmentation rectification method was adopted to optimize the SGM aggregation paths and reduce the propagation of incorrect paths. Experimental results demonstrate that the proposed method can effectively improve the stability and accuracy of dense matching, reduce the mismatching phenomenon, and finally produce high-quality disparity maps.

Meeting U.S. greenhouse gas emissions goals with the international air pollution provision of the clean air act

We explore economic, distributional and health consequences of U.S. greenhouse gas emissions objectives that could be achieved using Section 115 of the Clean Air Act (international air pollution), which has only recently received detailed legal analysis as a potential U.S. climate policy tool. Under it a national emissions target could be allocated among the states. This illustrative analysis considers 45% and 50% reductions of energy and industry-related CO2 emissions by 2030, below 2005 levels, via a model rule. Different approaches (based on legal precedent) for the interstate allocation are considered, along with alternative rates of technology improvement. The detail needed to analyze this approach is provided by MIT’s U.S. Regional Energy Policy model (30 individual states and multi-state regions), with its electricity sector replaced by the U.S. National Renewable Energy Laboratory’s Renewable Energy Development System (ReEDS). Air quality benefits are estimated using modeling tools developed by academic researchers and the U.S. Environmental Protection Agency. Three-quarters of emissions reductions in 2030 come in the electric sector, while reductions elsewhere illustrate the efficiency advantage of a multi-sector policy. With all states participating in allowance trading, the resulting national emissions price is lower than in older assessments. The difference is due to lower growth expectations, recent state policies, falling costs of low carbon technologies, and an improved representation of electric system flexibility by the ReEDS model. Even ignoring climate and air quality benefits, economic welfare grows at near the baseline rate for all regions regardless of the interstate allocation approach. When states distribute allowance revenue to residents on an equal per-capita basis, the policy is welfare improving to the lowest income quintile in all regions. Aggregation of control costs, the mortality effects of reduced particulates, and the value of avoided climate damages yields positive national net benefits in all cases.

Robust Scale-Aware Stereo Matching Network

Recently, deep convolutional neural networks (CNNs) have emerged as powerful tools for the correspondence problem in stereo matching task. However, the existence of multiscale objects and inevitable ill-conditioned regions, such as textureless regions, in real-world scene images continue to challenge current CNN architectures. In this article, we present a robust scale-aware stereo matching network, which aims to predict multiscale disparity maps and fuse them to achieve a more accurate disparity map. To this end, powerful feature representations are extracted from stereo images and are concatenated into a 4-D feature volume. The feature volume is then fed into a series of connected encoder–decoder cost aggregation structures for the construction of multiscale cost volumes. Following this, we regress multiscale disparity maps from the multiscale cost volumes and feed them into a fusion module to predict final disparity map. However, uncertainty estimations at each scale and complex disparity relationships among neighboring pixels pose a challenge on the disparity fusion. To overcome this challenge, we design a robust learning-based scale-aware disparity map fusion model, which seeks to map multiscale disparity maps onto the ground truth disparity map by leveraging their complementary strengths. Experimental results show that the proposed network is more robust and outperforms recent methods on standard stereo evaluation benchmarks.

A novel cell structure‐based disparity estimation for unsupervised stereo matching

It is well known that preserving depth edges is an effective solution for achieving the accurate disparity map in stereo matching, but many state-of-the-art methods do not preserve depth edges well. In order to solve it efficiently, the cell structure containing irregular and regular shape regions is designed to preserve depth edges. Based on the well-designed cell structure, a novel disparity estimation method for stereo matching is proposed, in which a two-layer disparity optimization method is proposed to refine the disparity plane; it includes the front-parallel disparities computation and slanted-surfaces disparity plane refinement. In the framework of front-parallel disparities computation, a tree-based cost aggregation method is presented to make full use of the segmentation information of cells and then performing semi-global cost aggregation. In the framework of slanted-surfaces disparity plane refinement, a new probability model is proposed that employs Bayesian inference for refining disparities in textureless, weak texture and occluded regions. Experimental results show that higher accuracy could be achieved via the proposed method compared with some known state-of-the-art stereo methods on KITTI 2015 and Middlebury dataset, which are the standard benchmarks for testing the stereo matching methods. It can also be indicated that the proposed method can produce accurate disparity map and have good generalization performance.

Development of depth map from stereo images using sum of absolute differences and edge filters

This article proposes a framework for the depth map reconstruction using stereo images. Fundamentally, this map provides an important information which commonly used in essential applications such as autonomous vehicle navigation, drone’s navigation and 3D surface reconstruction. To develop an accurate depth map, the framework must be robust against the challenging regions of low texture, plain color and repetitive pattern on the input stereo image. The development of this map requires several stages which starts with matching cost calculation, cost aggregation, optimization and refinement stage. Hence, this work develops a framework with sum of absolute difference (SAD) and the combination of two edge preserving filters to increase the robustness against the challenging regions. The SAD convolves using block matching technique to increase the efficiency of matching process on the low texture and plain color regions. Moreover, two edge preserving filters will increase the accuracy on the repetitive pattern region. The results show that the proposed method is accurate and capable to work with the challenging regions. The results are provided by the Middlebury standard dataset. The framework is also efficiently and can be applied on the 3D surface reconstruction. Moreover, this work is greatly competitive with previously available methods.

HLocalExp-CM: confidence map by hierarchical local expansion moves for accurate stereo matching

We present a stereo matching approach referred to as HLocalExp-CM by exploiting the hierarchical local contextual information and a confidence map based on a new grid structure. The proposed approach preserves fine depth edges and extracts accurate disparities in weak texture, textureless, and repeated texture regions. The proposed approach adopts a two-stage optimization strategy. In the framework of first stage, a multiresolution cost aggregation is minimized to reduce the search space of the disparity plane of each pixel. The second stage iteratively optimizes the confidence map and a global energy function to progressively improve the disparity accuracy for each pixel. The confidence map is estimated through classifying the pixels into distinctive and ambiguous ones by computing the decreasing rate of the multiresolution cost aggregation and then performs a spatial propagation and plane refinement for the update of the disparity of each pixel, thereby successfully eliminating the ambiguity of nondistinctive pixels. The global energy function based on a pairwise Markov random field uses cross-scale cost aggregation for taking advantage of context information of objects in different scenarios on local grid regions, which is different from the deep learning technique uses convolution layers extracting the context information. The proposed approach is evaluated on Middlebury benchmark V3, and is ranked first based on “bad 2.0 all metric,” a widely used criterion for the evaluation of stereo images, while the eighth place on “bad 2.0 nonocc metric” (recorded on July 24, 2021).

A 4.29nJ/pixel Stereo Depth Coprocessor With Pixel Level Pipeline and Region Optimized Semi-Global Matching for IoT Application

The semi-global matching (SGM) algorithm in stereo vision is a well-known depth-estimation method since it can generate dense and robust disparity maps. However, the real-time processing and low power dissipation, the specifications of the Internet-of-Thing (IoT) applications, are challenging for their computational complexity. In this paper, we propose a hardware-oriented SGM algorithm with pixel-level pipeline and region-optimized cost aggregation for high-speed processing and low hardware-resource usage. Firstly, the matching costs in a region are integrated with an optimization strategy to significantly reduce memory usage and improve the processing speed of the cost aggregation. Then, a two-layer parallel two-stage pipeline (TPTP) architecture, which enables pixel-level processing, is designed to calculate two directions (0° and 135°) aggregation to further solve the crucial computational bottleneck of the SGM algorithm. Finally, the architecture is demonstrated on a low-cost XILINX Spartan-7 device and an advanced Stratix-V FPGA device for VGA ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$640\times 480$ </tex-math></inline-formula> ) depth estimation. The experimental results show that the proposed architecture with compact hardware architecture also ensures accuracy. The pixel-level pipeline architecture enables a processing speed of 355 frames per second (fps) at 109MHz on the Spartan-7 FPGA device and 508 fps at 156MHz on the Stratix-V FPGA. Besides, the coprocessor respectively achieves an energy efficiency of 4.74 nJ/pixel with a power dissipation of 517mW and 4.29nJ/pixel with a power dissipation of 669mW on these two FPGAs.

To the application of two-stage dynamic programming in the problem of sequential visiting of megalopolises

It is considered the routing problem solution in which the task set is the union of two subsets (zones); one of subsets should be serviced earlier than the second. It is supposed that each task is connected with the visit to a megalopolis (nonempty finite set) and fulfilment of works. The regular succession of task fulfilment should satisfy to precedence conditions realized in each of zones. The cost functions assume a task list dependence; cost aggregation is supposed additive. For solution, the two-stage dynamic programming procedure is used.

Local PatchMatch Based on Superpixel Cut for Efficient High-resolution Stereo Matching

HIGHLIGHTS A new method of extracting the feature points in superpixels is proposed to compute the candidate disparities using non-local cost aggregation. A novel two-stage optimization framework for superpixel proposals by first getting the labels of feature points quickly and then consists the candidate label sets for updating the labels of pixels within the corresponding superpixel instead of assigning a label randomly to those pixels. The weight combination of intensity, gradient and binary image is designed for constructing an optimal minimum spanning tree to compute the aggregated matching cost.

Stereo Matching With Multiscale Hybrid Cost Volume

Stereo matching estimates the disparity between a pair of rectified left and right images, which plays an important role in robot navigation, autonomous driving, and other related tasks. Despite the remarkable progress of deep learning in stereo matching, there are still many challenges. One of the challenges is that current stereo models mostly generate a single scale cost volume based on costly 3D convolutions method or quick 2D convolutions method, but neither of these two methods can achieve a fair trade-off between quality and time. In this paper, we propose to construct multi-scale hybrid cost volume which aims at achieving fast speed while maintaining comparable accuracy. Concretely, we generate the correlation cost volume and the concatenation cost volume respectively, and then integrate together to form a hybrid cost volume which can significantly improve the accuracy and reduce the computational complexity. At the multi-scale level, we generate three hybrid cost volumes at different scales and then aggregate them by 2D convolutions which are faster than 3D convolutions. In addition, we adopt 2D CNN stacked hourglass with fused cost volume for cost aggregation. Specifically, the proposed method provides competitive performance with state-of-the-art methods, while being faster than most top-performing methods (e.g., 3.7× than PSMNet, 8.1× than GCNet, 16.3× than GANet). Extensive experiments of our model on the SceneFlow, KITTI 2012, and KITTI 2015 datasets demonstrate the competitiveness of our results.

A new function of stereo matching algorithm based on hybrid convolutional neural network

This paper proposes a new hybrid method between the learning-based and handcrafted methods for a stereo matching algorithm. The main purpose of the stereo matching algorithm is to produce a disparity map. This map is essential for many applications, including three-dimensional (3D) reconstruction. The raw disparity map computed by a convolutional neural network (CNN) is still prone to errors in the low texture region. The algorithm is set to improve the matching cost computation stage with hybrid CNN-based combined with truncated directional intensity computation. The difference in truncated directional intensity value is employed to decrease radiometric errors. The proposed method’s raw matching cost went through the cost aggregation step using the bilateral filter (BF) to improve accuracy. The winner-take-all (WTA) optimization uses the aggregated cost volume to produce an initial disparity map. Finally, a series of refinement processes enhance the initial disparity map for a more accurate final disparity map. This paper verified the performance of the algorithm using the Middlebury online stereo benchmarking system. The proposed algorithm achieves the objective of generating a more accurate and smooth disparity map with different depths at low texture regions through better matching cost quality.

Dense Feature Learning and Compact Cost Aggregation for Deep Stereo Matching

Recently, Convolutional Neural Networks (CNN) based deep models have been successfully applied to the task of stereo matching. In this paper, we propose a novel deep stereo matching network based on the strategies of dense feature learning and compact cost aggregation, namely DFL-CCA-Net. It consists of three modules: Dense Feature Learning (DFL), Compact Cost Aggregation (CCA) and the disparity regression module. In DFL module, the CNN backbone with Dense Atrous Spatial Pyramid Pooling (DenseASPP) is employed to extract multi-scale deep feature maps of the given left and right images respectively. Then an initial 4D cost volume is obtained by concatenating left feature maps with their corresponding right feature maps across each disparity level. In the following CCA module, each initial 3D cost volume component (i.e., the component across the left or right image feature channel dimension) is aggregated into a more compact one by using the atrous convolution operation with different expansion rates. These updated 3D cost volume components are then fed into the disparity regression module, which consisting of a 3D CNN network with a stacked hourglass structure, to estimate the final disparity map. Comprehensive experimental results demonstrated on the Scene Flow, KITTI 2012 and KITTI 2015 datasets show that the 3D cost volume components obtained by the proposed DFL and CCA modules generally containing more multi-scale semantic information and thus can largely improve the final disparity regression accuracies. Compared with other deep stereo matching methods, DFL-CCA-Net achieves very competitive prediction accuracies especially in the reflective regions and regions containing detail information.

Deep Stereo Matching With Hysteresis Attention and Supervised Cost Volume Construction.

Stereo matching disparity prediction for rectified image pairs is of great importance to many vision tasks such as depth sensing and autonomous driving. Previous work on the end-to-end unary trained networks follows the pipeline of feature extraction, cost volume construction, matching cost aggregation, and disparity regression. In this paper, we propose a deep neural network architecture for stereo matching aiming at improving the first and second stages of the matching pipeline. Specifically, we show a network design inspired by hysteresis comparator in the circuit as our attention mechanism. Our attention module is multiple-block and generates an attentive feature directly from the input. The cost volume is constructed in a supervised way. We try to use data-driven to find a good balance between informativeness and compactness of extracted feature maps. The proposed approach is evaluated on several benchmark datasets. Experimental results demonstrate that our method outperforms previous methods on SceneFlow, KITTI 2012, and KITTI 2015 datasets.

The first sub-meter resolution digital elevation model of the Kruger National Park, South Africa

The use of digital elevation models has proven to be crucial in numerous studies related to savanna ecosystem research. However, the insufficient spatial resolution of the chosen input data is often considered to be a limiting factor when conducting local to regional scale ecosystem analysis. The elevation models and orthorectified imagery created in this study represent the first wall-to-wall digital elevation data sets produced for the Kruger National Park (KNP), South Africa, at very high spatial resolution. Using colour-infrared (CIR) aerial imagery from the archives of the Chief Directorate: National Geo-spatial Information (CDNGI), Department of Agriculture, Land Reform and Rural Development (DALRRD) aerial acquisition programme, we created digital surface models (DSMs), digital terrain models (DTMs) and CIR orthomosaics covering the entire KNP with a nominal ground sampling distance of 0.25 m. Elevation information was derived using state-of-the-art stereo matching algorithms that utilised semi-global matching (SGM) as a cost aggregation function throughout the image pairing, using the Enterprise software from CATALYST. The final products were validated against reference products, and showed excellent agreement with R² values of 0.99. Further, the validation of the DTM and DSM revealed median absolute vertical height error (LE90) across all sites of 1.02 m and 2.58 m, respectively. The orthomosaics were validated with in situ ground control points (GCPs) exhibiting a horizontal Circular Probable Error (CPE) of 1.37 m. The data resulting from this work will be distributed freely with the aim of fostering more scientific studies in the African science community and beyond.Conservation implications: Accurate information about terrain and surface height are crucial inputs to a variety of scientific analysis, which are essential in protected areas, such as flood prediction or fire hazard estimation. Elevation data sets and orthomosaics in very high resolution can therefore serve as a crucial tool to improve park management and foster positive implications on conservation efforts.