Cost Aggregation Research Articles

Deep Neural Representation Guided Face Sketch Synthesis.

Face sketch synthesis shows great applications in a lot of fields such as online entertainment and suspects identification. Existing face sketch synthesis methods learn the patch-wise sketch style from the training dataset containing photo-sketch pairs. These methods manipulate the whole process directly in the field of RGB space, which unavoidably results in unsmooth noises at patch boundaries. If denoising methods are used, the sketch edges would be blurred and face structures could not be restored. Recent researches of feature maps, which are the outputs of a certain neural network layer, have achieved great success in texture synthesis and artistic image generation. In this paper, we reformulate the face sketch synthesis problem into a neural network feature maps based optimization task. Our results accurately capture the sketch drawing style and make full use of the whole stylistic information hidden in the training dataset. Unlike former feature map based methods, we utilize the Enhanced 3D PatchMatch and cross-layer cost aggregation methods to obtain the target feature maps for the final results. Multiple experiments have shown that our approach imitates hand-drawn sketch style vividly, and has high-quality visual effects on CUHK, AR, XM2VTS and CUFSF face sketch datasets.

A hierarchical stereo matching algorithm based on adaptive support region aggregation method

Stereo matching is a fundamental process in many application fields. An accurate depth information is useful for stereo systems to separate occluding image components. In the conception of stereo matching algorithms, various works rely on the Census Transform (CT) as a cost computation step, due to its robustness against radiometric changes. In this paper, we propose a new variant of the CT cost function which incorporates edge side information for further compensating radiometric changes. We demonstrate that the proposed variant matching cost improves significantly the quality of the disparity results. In addition, an aggregation method, based on the adaptive support region, is implemented into a hierarchical fusion processing scheme, by incorporating low frequency information into high frequency using a robust exponential function. This allows a multi-scale interaction within the adaptive cost aggregation. Hence, significant erroneous disparities are reduced, especially in textureless regions. Experiments were conducted on the KITTI benchmark and the obtained results have shown the potential merits of the proposed framework.

Deep self-guided cost aggregation for stereo matching

In this paper, we present a deep self-guided cost aggregation method used to obtain an accurate disparity map from a pair of stereo images. Conventional cost aggregation methods typically perform joint image filtering on each cost volume slice. Thus, a guidance image is necessary for the conventional methods to work effectively. However, a guidance image might be unreliable due to several distortions, such as noise, blur, radiometric variation. Based on our observations, each cost volume slice can guide itself based on the internal features. However, finding a direct mapping function from the initial and filtered cost volume slice without any guidance image is difficult. To solve this problem, we use an advanced deep learning technique to perform self-guided cost aggregation. Because of the absence of ground truth cost volume, we offer the solution for the dataset generation. Our proposed deep learning network consists of two sub-networks: dynamic weight network and descending filtering network. We integrate the feature reconstruction loss and the pixelwise mean square loss function to preserve the edge property. Experimental results show that the proposed method achieves better results even though it does not employ a guidance image.

Revisiting dynamic programming for precedence-constrained traveling salesman problem and its time-dependent generalization

The precedence constrained traveling salesman problem (TSP-PC), or the sequential ordering problem (SOP), consists of finding an optimal TSP tour that will also satisfy the namesake precedence constraints, typically specified as a partial order or a directed acyclic graph. Its dynamic programming (DP) solution was proposed as early as 1979, however, by late 1990s, it mostly fell out of use in plain TSP-PC. Revisiting this method, we are able to close one of the long-standing TSPLIB SOP problem instances, ry48p.3.sop, and provide improved bounds on its time complexity. Harnessing the “omnivorous” nature of DP, we prove the validity of DP optimality principle for TSP-PC with both (i) abstract cost aggregation function, which may be the arithmetic + operation as in “ordinary” TSP or max as in Bottleneck TSP, or any other left-associative nondecreasing in the first argument operation and (ii) travel cost functions depending on the set of pending tasks (“sequence dependence”). Using the latter generalization, we close several TD-SOP (time-dependent TSP-PC) instances based on TSPLIB SOP as proposed by Kinable et al., including rbg253a.sop. Through the restricted DP heuristic, which was originally formulated for time-dependent TSP by Malandraki and Dial, we improve the state-of-the-art upper bounds for all yet unsolved TSPLIB-based TD-SOP instances, including those with more than 100 cities. We also improve worst-case complexity estimates for DP in TSP-PC.

STEREO IMAGE DENSE MATCHING BY INTEGRATING SIFT AND SGM ALGORITHM

Abstract. Semi-global matching(SGM) performs the dynamic programming by treating the different path directions equally. It does not consider the impact of different path directions on cost aggregation, and with the expansion of the disparity search range, the accuracy and efficiency of the algorithm drastically decrease. This paper presents a dense matching algorithm by integrating SIFT and SGM. It takes the successful matching pairs matched by SIFT as control points to direct the path in dynamic programming with truncating error propagation. Besides, matching accuracy can be improved by using the gradient direction of the detected feature points to modify the weights of the paths in different directions. The experimental results based on Middlebury stereo data sets and CE-3 lunar data sets demonstrate that the proposed algorithm can effectively cut off the error propagation, reduce disparity search range and improve matching accuracy.

Stereo Matching via Dual Fusion

We present a two-fold fusion framework that constructs comprehensive cost volumes for stereo matching. To this end, we develop fusion schemes at two key steps, i.e., a) the raw cost computation and b) the cost aggregation. Specifically, we commence by fusing both structure- and data-oriented features as raw costs. We then incorporate the guided filtered costs into the cross-based cost aggregation and obtain the fused aggregated costs. The fusion schemes at the two steps effectively complement each other and result in an accurate disparity map. Experiments on the Middlebury benchmark v3 demonstrate the state-of-the-art performance of our framework in terms of various metrics.

Dense Stereo Matching With Edge-Constrained Penalty Tuning

Dense stereo matching has been an indispensable technique for 3-D exploitation of imagery, in which semiglobal matching (SGM) is one of the outstanding approaches and has numerous variations in the literature. However, generating accurate disparity in depth discontinuity areas remains challenging, since the cost-aggregation step of SGM providing consistency constraints is sensitive to nonoptimal parametrizations of relevant penalty functions. In this letter, we present a gradual SGM cost aggregation that comprises a penalty tuning process and involves edge feature knowledge. To be more specific, we propose an additional penalty parameter and a weighting formula to handle edge pixels with depth variations, acquiring satisfactory depth estimation by preserving sharp geometric edges and maintaining smoothness without raising extra noise. Evaluations on well-known benchmark and real large-scale images are promising, and the latter case yields point clouds with sharp edges and smooth surfaces showing the effectiveness and feasibility of the proposed method.

Deep Stereo Matching With Explicit Cost Aggregation Sub-Architecture

Deep neural networks have shown excellent performance for stereo matching. Many efforts focus on the feature extraction and similarity measurement of the matching cost computation step while less attention is paid on cost aggregation which is crucial for stereo matching. In this paper, we present a learning-based cost aggregation method for stereo matching by a novel sub-architecture in the end-to-end trainable pipeline. We reformulate the cost aggregation as a learning process of the generation and selection of cost aggregation proposals which indicate the possible cost aggregation results. The cost aggregation sub-architecture is realized by a two-stream network: one for the generation of cost aggregation proposals, the other for the selection of the proposals. The criterion for the selection is determined by the low-level structure information obtained from a light convolutional network. The two-stream network offers a global view guidance for the cost aggregation to rectify the mismatching value stemming from the limited view of the matching cost computation. The comprehensive experiments on challenge datasets such as KITTI and Scene Flow show that our method outperforms the state-of-the-art methods.

Efficient cost aggregation for feature-vector-based wide-baseline stereo matching

■■■In stereo matching applications, local cost aggregation techniques are usually preferred over global methods due to their speed and ease of implementation. Local methods make implicit smoothness assumptions by aggregating costs within a finite window; however, cost aggregation is a time-consuming process. Furthermore, most existing local methods are based on pixel intensity values, and hence are not efficient with feature vectors used in wide-baseline stereo matching. In this paper, a new cost aggregation method is proposed, where a Per-Column Cost matrix is combined with a feature-vector-based weighting strategy to achieve both matching accuracy and computational efficiency. Here, the proposed cost aggregation method is applied with the DAISY feature descriptor for wide-baseline stereo matching; however, this method can also be applied to a fast growing number of stereo matching techniques that are based on feature descriptors. A performance comparison with several benchmark local cost aggregation approaches is presented, along with a thorough analysis of the time and storage complexity of the proposed method.

Approaches to Aggregation and Decision Making—A Health Economics Approach: An ISPOR Special Task Force Report [5

The fifth section of our Special Task Force report identifies and discusses two aggregation issues: 1) aggregation of cost and benefit information across individuals to a population level for benefit plan decision making and 2) combining multiple elements of value into a single value metric for individuals. First, we argue that additional elements could be included in measures of value, but such elements have not generally been included in measures of quality-adjusted life-years. For example, we describe a recently developed extended cost-effectiveness analysis (ECEA) that provides a good example of how to use a broader concept of utility. ECEA adds two features—measures of financial risk protection and income distributional consequences. We then discuss a further option for expanding this approach—augmented CEA, which can introduce many value measures. Neither of these approaches, however, provide a comprehensive measure of value. To resolve this issue, we review a technique called multicriteria decision analysis that can provide a comprehensive measure of value. We then discuss budget-setting and prioritization using multicriteria decision analysis, issues not yet fully resolved. Next, we discuss deliberative processes, which represent another important approach for population- or plan-level decisions used by many health technology assessment bodies. These use quantitative information on CEA and other elements, but the group decisions are reached by a deliberative voting process. Finally, we briefly discuss the use of stated preference methods for developing “hedonic” value frameworks, and conclude with some recommendations in this area.

Improving stereo matching algorithm with adaptive cross-scale cost aggregation

Human beings process stereoscopic correspondence across multiple purposes like robot navigation, automatic driving, and virtual or augmented reality. However, this bioinspiration is ignored by state-of-the-art dense stereo correspondence matching methods. Cost aggregation is one of the critical steps in the stereo matching method. In this article, we propose an optimized cross-scale cost aggregation scheme with coarse-to-fine strategy for stereo matching. This scheme implements cross-scale cost aggregation with the smoothness constraint on neighborhood cost, which essentially extends the idea of the inter-scale and intra-scale consistency constraints to increase the matching accuracy. The neighborhood costs are not only used in the intra-scale consistency to ensure that the regularized costs vary smoothly in an eight-connected neighbors region but also incorporated with inter-scale consistency to optimize the disparity estimation. Additionally, the improved method introduces an adaptive scheme in each scale with different aggregation methods. Finally, experimental results evaluated both on classic Middlebury and Middlebury 2014 data sets show that the proposed method performs better than the cross-scale cost aggregation. The whole stereo correspondence algorithm achieves competitive performance in terms of both matching accuracy and computational efficiency. An extensive comparison, including the KITTI benchmark, illustrates the better performance of the proposed method also.

Real-Time Stereo Vision System: A Multi-Block Matching on GPU

Real-time stereo vision is attractive in many areas such as outdoor mapping and navigation. As a popular accelerator in the image processing field, GPU is widely used for the studies of the stereo vision algorithms. Recently, many stereo vision systems on GPU have achieved low error rate, as a result of the development of deep learning. However, their processing speed is normally far from the real-time requirement. In this paper, we propose a real-time stereo vision system on GPU for the high-resolution images. This system also maintains a low error rate compared with other fast systems. In our approach, the image is resized to reduce the computational complexity and to realize the real-time processing. The low error rate is kept by using the cost aggregation with multiple blocks, secondary matching and sub-pixel estimation. Its processing speed is 41 fps for $2888\times 1920$ pixels images when the maximum disparity is 760.

Second-Order Semi-Global Stereo Matching Algorithm Based on Slanted Plane Iterative Optimization

Stereo matching is a difficult and challenging task due to many uncontrollable factors that affect the results. These factors include the non-ideal radiometric conditions and the presence of weak-textured regions. A stereo matching algorithm including cost computation, cost aggregation, disparity computation, and disparity refinement is proposed to overcome these limitations. First, census, gradient, and HSL measures are combined as the pixel-wise matching cost to reduce the radiometric distortions. Second, a second-order smoothness constraint based on angle direction priors is utilized to improve the matching accuracy in the weak-textured regions. Then, the matching cost and smoothness constraint are applied to semi-global matching. Third, a winner-takes-all strategy is adopted to calculate the initial disparity maps. Finally, a coarse-to-fine simple linear iterative clustering superpixel segmentation is proposed to split the image into some regions with similar colors, adjacent pixels, and similar disparity planes. Through the segmentation result, boundary lines are classified into occluded, hinge, and coplanar lines. Slanted plane iterative optimization is performed to obtain the optimized disparity maps according to the segmentation result and boundary lines classification. The algorithm is evaluated in the Middlebury and KITTI datasets. The experimental results show that the method has good performance under non-ideal radiometric conditions and in the weak-textured regions. The disparity maps obtained by the method have a lower mismatching rate compared with other algorithms.

Multiscale Feature Extractors for Stereo Matching Cost Computation

We propose four efficient feature extractors based on convolutional neural networks for stereo matching cost computation. Two of them generate multiscale features with diverse receptive field sizes. These multiscale features are used to compute the corresponding multiscale matching costs. We then determine an optimal cost by combining the multiscale costs using edge information. On the other hand, the other two feature extractors produce uni-scale features by combining multiscale features directly through fully connected layers. Finally, after obtaining matching costs using one of the four extractors, we determine optimal disparities based on the cross-based cost aggregation and the semiglobal matching. Extensive experiments on the Middlebury stereo data sets demonstrate the effectiveness and efficiency of the proposed algorithm. Specifically, the proposed algorithm provides competitive matching performance with the state of the arts, while demanding lower computational complexity.

Image Super-Resolution Reconstruction Based on Disparity Map and CNN

This paper presents a new method for registration of digital micro-mirror device (DMD) cameras through image reconstruction. The method called dual-channel super-resolution reconstruction (DCSR) uses a dual-channel input convolutional neural network method to iteratively obtain high resolution images. The originality of this study relies on the input color weight disparity map that enables extracting edges cues and feature cues effectively during the super-resolution. The iterative initial disparity map is obtained by a cost aggregation method based on the color weights, and the model parameters are updated during the iterative process. To validate performance, a series of the comparison experiments is developed on different network structures and parameter settings for the diverse deconv-pooling layers, convolution kernel sizes, and activation function. Experimental results show that the DCSR method can solve the sub-pixel registration problem of DMD cameras and achieve a good image reconstruction effect. The DCSR performs higher accuracy and an efficient process compared with traditional reconstruction made by single-channel input convolution neural network. The spatial resolution of the reconstructed image is significantly improved, and the detail resolution is enhanced.

基于水平树结构的可变权重代价聚合立体匹配算法

基于水平树结构的可变权重代价聚合立体匹配算法

Cost aggregation in export logistics chain

This paper analyses cost aggregation in a supply chain. It provides a literature overview on the key concepts of cost aggregation, multimodal transport, logistic chain and maritime transport. The focuses on the value adding process with logistics data and assesses the costs accumulation during the transport process. The paper also reveals multimodal impacts on the logistics costs. The research data is obtained from the costing system of a large export company. The company exports round 90% of its production and mostly to the European markets. The research data contains a sample of 929 invoiced orders to the largest market of the mill. The research results indicate empirical evidence of the cost-function properties. This type of an approach is rare in logistics literature as these detailed data sets are highly difficult to obtain. The article concludes by addressing future research task and directions.

SVCV: segmentation volume combined with cost volume for stereo matching

Stereo matching between binocular stereo images is fundamental to many computer vision tasks, such as three‐dimensional (3D) reconstruction and robot navigation. Various structures of real 3D scenes lead stereo matching to be an old yet still challenging problem. In this study, the authors proposed a novel adaptive support weights technique which exploits the hierarchical information provided by multilevel segmentation to preserve the robustness to imaging conditions and spatial proximity in cost aggregation. Besides, a generalisable cost refinement strategy is designed to remove the matching ambiguity in large weakly textured regions. The proposed strategy utilises both the fluctuation of the filtered cost volume and the colour information to further improve the matching accuracy. Experimental results of 50 stereo images demonstrate the effectiveness and efficiency of the proposed method. Furthermore, a systematic evaluation is developed to assess the conventional steps in local stereo methods and then reliable suggestions are given to the beginners and researchers outside the stereo matching field.

스테레오 영상 기반의 객체 탐지 및 객체의 3차원 위치 추정

본 항공기에 스테레오 카메라를 장착하여 영상 기반의 비행 객체 탐지 및 탐지된 객체의 3차원 위치를 추정하는 방법을 제시하였다. 구름 사이에 존재할 수 있는 원거리의 작은 객체를 탐지하기 위한 방법으로 PCT 기반의 Saliency Map을 생성하여 이용하였으며, 이렇게 탐지된 객체는 좌우 스테레오 영상에서 매칭을 수행하여 스테레오 시차(Disparity)를 추출하였다. 정확한 Disparity를 추출하기 위하여 비용집적(Cost Aggregation) 영역을 탐지 객체에 맞추어 가변되도록 가변 영역으로 사용하였으며, 본 논문에서는 Saliency Map에서 객체의 존재 영역으로 검출된 결과를 사용하였다. 좀 더 정밀한 Disparity를 추출하기 위하여 Sub-pixel interpolation 기법을 사용하여 Sub-pixel 레벨의 실수형 Disparity를 추출하였다. 또한 이에 카메라 파라미터를 적용하여 실제 탐지된 비행 객체의 3차원 공간 좌표를 생성하여 객체의 공간위치를 추정하는 방법을 제시하였다. 이는 향후 자율비행체의 영상기반 객체 탐지 및 충돌방지 시스템에 활용될 수 있을 것으로 기대된다. We introduced a stereo camera on the aircraft to detect flight objects and to estimate the 3D position of them. The Saliency map algorithm based on PCT was proposed to detect a small object between clouds, and then we processed a stereo matching algorithm to find out the disparity between the left and right camera. In order to extract accurate disparity, cost aggregation region was used as a variable region to adapt to detection object. In this paper, we use the detection result as the cost aggregation region. In order to extract more precise disparity, sub-pixel interpolation is used to extract float type-disparity at sub-pixel level. We also proposed a method to estimate the spatial position of an object by using camera parameters. It is expected that it can be applied to image - based object detection and collision avoidance system of autonomous aircraft in the future.

A local stereo matching algorithm based on weighted guided image filtering for improving the generation of depth range images

Stereo matching is a challenging problem and highly accurate depth image is important in different applications. The main problem is to estimate the correspondence between two pixels in a stereo pair. To solve this problem, in the last decade, several cost aggregation methods aimed at improving the quality of stereo matching algorithms have been introduced. We propose a new cost aggregation method based on weighted guided image filtering (WGIF) for local stereo matching. The proposed algorithm solves multi-label problems in three steps. First, the cost volume is constructed using pixel-wise matching cost computation functions. Then, each slice of the cost volume is independently filtered using the WGIF, which substitutes for the smoothness term in the energy function. Finally, the disparity of any pixel is simply computed. The WGIF uses local weights based on a variance window of pixels in a guidance image for cost volume filtering. Experimental results using Middlebury stereo benchmark verify that the proposed method is effective due to a high quality cost volume filter.