Matching Window Size Research Articles

MVP-Stereo: A Parallel Multi-View Patchmatch Stereo Method with Dilation Matching for Photogrammetric Application

Multi-view stereo plays an important role in 3D reconstruction but suffers from low reconstruction efficiency and has difficulties reconstructing areas with low or repeated textures. To address this, we propose MVP-Stereo, a novel multi-view parallel patchmatch stereo method. MVP-Stereo employs two key techniques. First, MVP-Stereo utilizes multi-view dilated ZNCC to handle low texture and repeated texture by dynamically adjusting the matching window size based on image variance and using a portion of pixels to calculate matching costs without increasing computational complexity. Second, MVP-Stereo leverages multi-scale parallel patchmatch to reconstruct the depth map for each image in a highly efficient manner, which is implemented by CUDA with random initialization, multi-scale parallel spatial propagation, random refinement, and the coarse-to-fine strategy. Experiments on the Strecha dataset, the ETH3D benchmark, and the UAV dataset demonstrate that MVP-Stereo can achieve competitive reconstruction quality compared to state-of-the-art methods with the highest reconstruction efficiency. For example, MVP-Stereo outperforms COLMAP in reconstruction quality by around 30% of reconstruction time, and achieves around 90% of the quality of ACMMP and SD-MVS in only around 20% of the time. In summary, MVP-Stereo can efficiently reconstruct high-quality point clouds and meet the requirements of several photogrammetric applications, such as emergency relief, infrastructure inspection, and environmental monitoring.

Polarimetric SAR pixel offset tracking for large-gradient landslide displacement mapping

Interferometry and pixel offset tracking (POT) are two major technologies to measure ground deformation with repeat-pass synthetic aperture radar (SAR) observations. The SAR POT method can tolerate a much larger deformation gradient than interferometric SAR (InSAR), and thus it has been widely applied in monitoring earthquakes, volcanoes, landslides, glaciers, etc. However, there is an intrinsic trade-off between matching accuracy and detail preserving in terms of matching window size for the traditional SAR POT method using only single-polarization intensity. Taking advantage of polarimetric scattering properties, this study develops a novel POT method, named PCM-POT, towards mitigating the dependence of offset estimations on matching window size. It employs a generalized likelihood ratio test (GLRT) upon polarimetric covariance matrices (PCMs) to evaluate the similarity of two candidate pixels and estimate high-resolution surface displacements with high accuracy. The effectiveness and advantage of PCM-POT are demonstrated by an experiment of retrieving the displacements of the Slumgullion landslide from a pair of full-polarimetric UAVSAR images. Comparisons with the traditional POT indicate that using polarimetric scattering properties can reduce the uncertainty in offset estimations over heterogeneous sliding areas. Further experiments suggest that PCM-POT is also suitable for dual-polarimetric satellite SAR data.

Effect of the Matching Window Size and TDI Stage Number on Image-Based Satellite Jitter Detection

For time-delay integration (TDI) linear push-broom satellite images, it is generally considered that the matching window size and TDI stage number affect the jitter detection capability, but there are very few quantitative studies. In this letter, a scheme for investigating the effect of the matching window size and TDI stage number on image-based jitter detection is designed, and the model of image motion caused by jitter and the model of jitter detection and estimation for TDI images are deduced and established. Through a comprehensive experimental analysis, it is revealed that the influence of the TDI stage number and matching window size on jitter frequency detection is not significant, which improves our previous understanding. Although the TDI mode will attenuate the jitter amplitude by integration, the real jitter amplitude can be accurately retrieved by the proposed estimation model with attenuation compensation. The matching window size in the row direction has a significant effect on jitter amplitude detection. To ensure the detection accuracy of the jitter amplitude, the matching window size in the row direction is suggested to be less than 1/5 of the image line count corresponding to a jitter cycle.

Change Detection in SAR Images via Ratio-Based Gaussian Kernel and Nonlocal Theory

Compared with the synthetic aperture radar (SAR) image processing theory based on local neighborhood, the nonlocal theory is not limited to a local neighborhood of an image and has great potential in change detection of SAR images. In this study, an approach using ratio-based nonlocal information (RNLI) is proposed for change detection in multitemporal SAR images. First, the RNLI is extracted from a spatial–temporal nonlocal neighborhood where the similarity of two pixels in the nonlocal neighborhood is well characterized by the proposed ratio-based Gaussian kernel function. The parameters of RNLI: noise level and matching window size are adaptively determined to avoid the uncertainty of the change detection result caused by user experience. Second, the difference image is generated by using the RNLI and the ratio operator. Finally, the change map is obtained by segmenting the difference image with a threshold. Experiments conducted on two real datasets and two simulated datasets showed that the proposed method performed better than the other advanced change detection methods, which can better retain the edge information of the changed area while reducing the overall error of the change detection results.

Spatiotemporal Correlation-Based Accurate 3D Face Imaging Using Speckle Projection and Real-Time Improvement

The reconstruction of 3D face data is widely used in the fields of biometric recognition and virtual reality. However, the rapid acquisition of 3D data is plagued by reconstruction accuracy, slow speed, excessive scenes and contemporary reconstruction-technology. To solve this problem, an accurate 3D face-imaging implementation framework based on coarse-to-fine spatiotemporal correlation is designed, improving the spatiotemporal correlation stereo matching process and accelerating the processing using a spatiotemporal box filter. The reliability of the reconstruction parameters is further verified in order to resolve the contention between the measurement accuracy and time cost. A binocular 3D data acquisition device with a rotary speckle projector is used to continuously and synchronously acquire an infrared speckle stereo image sequence for reconstructing an accurate 3D face model. Based on the face mask data obtained by the high-precision industrial 3D scanner, the relationship between the number of projected speckle patterns, the matching window size, the reconstruction accuracy and the time cost is quantitatively analysed. An optimal combination of parameters is used to achieve a balance between reconstruction speed and accuracy. Thus, to overcome the problem of a long acquisition time caused by the switching of the rotary speckle pattern, a compact 3D face acquisition device using a fixed three-speckle projector is designed. Using the optimal combination parameters of the three speckles, the parallel pipeline strategy is adopted in each core processing unit to maximise system resource utilisation and data throughput. The most time-consuming spatiotemporal correlation stereo matching activity was accelerated by the graphical processing unit. The results show that the system achieves real-time image acquisition, as well as 3D face reconstruction, while maintaining acceptable systematic precision.

BRIEF-based face descriptor: an application to automatic facial expression recognition (AFER)

This paper presents a new face descriptor based on binary robust independent elementary features (BRIEF) (Calonder et al. in IEEE Trans Pattern Anal Mach Intell 34(7):1281–1298, 2012; in: European conference on computer vision, Springer, pp 778–792, 2010). The most important properties of BRIEF are the independence of the descriptor length from the matching window size and the possibility of using a subsample of pixel pairs located at arbitrary positions in the matching window. Furthermore, BRIEF is computationally simple and gives a compact representation. The BRIEF descriptor can be used to generate discriminative features globally from an image. However, when BRIEF is used to generate features from a region of an image with no explicit shape such as a face in an image, BRIEF must be used locally to ensure that each pixel in the region is evaluated locally to capture the local properties. In this paper, the performance of BRIEF feature is evaluated in the task of AFER. Using three different facial expression databases, we demonstrate that BRIEF provides satisfactory, encouraging and comparable performance to the performance of alternative face representation tested on the same data, and it is efficient in terms of memory usage and execution time.

EFFECT OF IMAGE MATCHING WINDOW SIZE ON SATELLITE JITTER FREQUENCY DETECTION

Abstract. Satellite attitude jitter is a common and complex phenomenon for high-resolution satellites and it is detectable by multi-temporal image matching. This paper analyses the effect of image matching window size on jitter frequency detection. First, two simulation images with a given short time lag and line scanning frequency affected by a modelled jitter are generated based on the principle of dynamic imaging model. Then, the relative image distortions are obtained by dense image matching with different matching window size and the frequency is estimated through spectrum analysis of the obtained image distortions. The experimental results demonstrated the feasibility and reliability of high frequency jitter detection based on dense image matching, and the results indicated that the maximum detectable frequency is almost not affected by the change of image matching window size, which provided useful demonstration of image-based satellite jitter detection capacity.

Evaluation of Matching Costs for High-Quality Sea-Ice Surface Reconstruction from Aerial Images

Satellite remote sensing can be used effectively with a wide coverage and repeatability in large-scale Arctic sea-ice analysis. To produce reliable sea-ice information, satellite remote-sensing methods should be established and validated using accurate field data, but obtaining field data on Arctic sea-ice is very difficult due to limited accessibility. In this situation, digital surface models derived from aerial images can be a good alternative to topographical field data. However, to achieve this, we should discuss an additional issue, i.e., that low-textured surfaces on sea-ice can reduce the matching accuracy of aerial images. The matching performance is dependent on the matching cost and search window size used. Therefore, in order to generate high-quality sea-ice surface models, we first need to examine the influence of matching costs and search window sizes on the matching performance on low-textured sea-ice surfaces. For this reason, in this study, we evaluate the performance of matching costs in relation to changes of the search window size, using acquired aerial images of Arctic sea-ice. The evaluation concerns three factors. The first is the robustness of matching to low-textured surfaces. Matching costs for generating sea-ice surface models should have a high discriminatory power on low-textured surfaces, even with small search windows. To evaluate this, we analyze the accuracy, uncertainty, and optimal window size in terms of template matching. The second is the robustness of positioning to low-textured surfaces. One of the purposes of image matching is to determine the positions of object points that constitute digital surface models. From this point of view, we analyze the accuracy and uncertainty in terms of positioning object points. The last is the processing speed. Since the computation complexity is also an important performance indicator, we analyze the elapsed time for each of the processing steps. The evaluation results showed that the image domain costs were more effective for low-textured surfaces than the frequency domain costs. In terms of matching robustness, the image domain costs showed a better performance, even with smaller search windows. In terms of positioning robustness, the image domain costs also performed better because of the lower uncertainty. Lastly, in terms of processing speed, the PC (phase correlation) of the frequency domain showed the best performance, but the image domain costs, except MI (mutual information), were not far behind. From the evaluation results, we concluded that, among the compared matching costs, ZNCC (zero-mean normalized cross-correlation) is the most effective for sea-ice surface model generation. In addition, we found that it is necessary to adjust search window sizes properly, according to the number of textures required for reliable image matching on sea-ice surfaces, and that various uncertainties due to low-textured surfaces should be considered to determine the positions of object points.

Binary descriptor-based dense line-scan stereo matching

We present a line-scan stereo system and descriptor-based dense stereo matching for high-performance vision applications. The stochastic binary local descriptor (STABLE) descriptor is a local binary descriptor that builds upon the principles of compressed sensing theory. The most important properties of STABLE are the independence of the descriptor length from the matching window size and the possibility that more than one pair of pixels contributes to a single-descriptor bit. Individual descriptor bits are computed by comparing image intensities over pairs of balanced random subsets of pixels chosen from the whole described area. On a synthetic as well as real-world examples, we demonstrate that STABLE provides competitive or superior performance than other state-of-the-art local binary descriptors in the task of dense stereo matching. The real-world example is derived from line-scan binocular stereo imaging, i.e., two line-scan cameras are observing the same object line and 2-D images are generated due to relative motion. We show that STABLE performs significantly better than the census transform and local binary patterns (LBP) in all considered geometric and radiometric distortion categories to be expected in practical applications of stereo vision. Moreover, we show as well that STABLE provides comparable or better matching quality than the binary robust-independent elementary features descriptor. The low computational complexity and flexible memory footprint make STABLE well suited for most hardware architectures. We present quantitative results based on the Middlebury stereo dataset as well as illustrative results for road surface reconstruction.

Stereo-camera-based object detection using fuzzy color histograms and a fuzzy classifier with depth and shape estimations

This paper proposes a new method of detecting an object containing multiple colors with non-homogeneous distributions in complex backgrounds and subsequently estimating the depth and shape of the object using a stereo camera. To extract features for object detection, this paper proposes fuzzy color histograms (FCHs) based on the self-splitting clustering (SSC) of the hue-saturation (HS) color space. For each scanning window in a pyramid of scaled images, the FCH is obtained by accumulating the fuzzy degrees of all of the pixels belonging to each cluster. The FCH is fed to a fuzzy classifier to detect an object in the left image captured by the stereo camera. To find the matched object region in the right image, the left and right images are first segmented using the SSC-partitioned HS space. The depth of the object is then found by performing stereo matching on the segmented images. To find the shape of the object, a disparity map is built using the estimated object depth to automatically determine the stereo matching window size and disparity search range. Finally, the shape of the object is segmented from the disparity map. The experimental results of the detection of different objects with depth and shape estimations are used to verify the performance of the proposed method. Comparisons with different detection and disparity map construction methods are performed to demonstrate the advantage of the proposed method.

Rear obstacle detection system with fisheye stereo camera using HCT

A vision based rear obstacle detection system is one of the most essential technologies, which can be used in many applications such as a parking assistance systems and intelligent vehicles. Stereo matching based obstacle detection methods have two inherent limitations, including sensitivity to illumination variation and high computational complexity. To overcome these problems, we present a hierarchical census transform (HCT)-based stereo matching method, and proposes a real-time rear obstacle detection system using fisheye stereo cameras. It includes two major components: firstly, we use a hierarchical approach to improve computational efficiency and reduce dependency on a matching window size. Computation time can also be accelerated with compute unified device architecture (CUDA) implementation because we designed the proposed method for parallel processing. Secondly, our method provides more accurate depth information, especially for moving objects, through cost aggregation using color and motion information. Experimental results show satisfactory performance under various real parking environments. The performance of the proposed method is superior to those of the conventional methods in terms of runtime and accuracy of depth estimation. Moreover, correct detection rate of the proposed system is 12.51% better than those of other systems, while its false and miss detection rates are 11.09% and 12.51% lower than those of other detection systems.

Hyperspectral Remote Sensing Data Recovery via Adaptive Window Matching Method

HJ-1A satellite is often used to monitor environmental disaster and plays an important role in environmental changes. Because of the affection of various factors, certain band of HJ-1A hyperspectral remote sensing data is severe loss or distortion, which brings great difficulties for subsequent quantitative processing. A novel adaptive window matching algorithm, which can adjust intelligently size of matching window according to different local feature information of the image, is proposed for HJ-1A satellite hyperspectral data recovery in this paper. The results show that the adaptive matching algorithm has a more superior performance than other algorithms in image quality index, column mean curve, and image correlation coefficient.

Three-Dimensional Information Acquisition Based on Artificial Compound Eye Vision System

In this paper, the artificial compound eye vision system is used to develop an effective algorithm for acquiring 3D information of object. According to compound eye vision system constructed, the imaging principle and correlation between the object point and the imaging point is analyzed. Then the cross correlation matching algorithm is present to extract the depth and obtain the 3D information. The influence factors such as number of matched elemental images and matching window size are considered and analyzed. The experimental is implemented to verify the validity of the proposed method.

Dense Stereo Matching Method Based on Local Affine Model

A new method for constructing an accurate disparity space image and performing an efficient cost aggregation in stereo matching based on local affine model is proposed in this paper. The key algorithm includes a new self-adapting dissimilarity measurement used for calculating the matching cost and a local affine model used in cost aggregation stage. Different from the traditional region-based methods, which try to change the matching window size or to calculate an adaptive weight to do the aggregation, the proposed method focuses on obtaining the efficient and accurate local affine model to aggregate the cost volume while preserving the disparity discontinuity. Moreover, the local affine model can be extended to the color space. Experimental results demonstrate that the proposed method is able to provide subpixel precision disparity maps compared with some state-of-the-art stereo matching methods.

TRAINING BASED ON REAL-TIME MOTION EVALUATION FOR FUNCTIONAL REHABILITATION IN VIRTUAL ENVIRONMENT

One of the most effective applications of virtual reality (VR) in physical rehabilitation is training, where patients are trained for sequence decision-making in special situations presented in virtual environment. In this application, the evaluation of the movement of the subject performing a physical task is crucial. A good evaluation of the motion is necessary to follow the progression of the patient during his training session. Therefore, it helps therapist to better supervise therapeutic planning. Actually, the performance of the patient's training is determined by subjective observation of the therapist. Our approach is to propose a system that allows the patient to perform his training and to evaluate the progress of training in an autonomous way. This system consists of a motion analysis technique for a rehabilitation application where the patient is represented by his own avatar in virtual environment. The task performance required from the patient is his capability to reproduce in real time a movement. The real-time motion evaluation technique is based on the time series data matching method called Longest Common Sub-Sequence (hereafter LCSS). It is used to calculate distance between the reference motion of virtual avatar and the captured motion data of the patients and thus is used to determine how well the patients are doing during the training. The complexity of the technique proposed is in the order of O (δ) in which δ is a constant matching window size. Our prototype application is based on Tai-chi movements which have shown many health benefits and are increasingly used for therapeutic purposes.

New disparity estimation scheme based on adaptive matching windows for intermediate view reconstruction

A new intermediate view reconstruction technique based on the adaptive disparity estimation algorithm is proposed. First, the feature values indicating the local image complexity are extracted from the input stereo image pair by using edge detection and matching algorithms. Then, the matching window size for disparity estimation is adaptively selected depending on the magnitude of these feature values. That is, for the region having larger feature values, the smaller matching window size is selected, while for the opposite case, the larger matching window size is selected by comparison with a predetermined threshold value. This new approach is not only able to reduce the mismatch of the disparity vector, which occurs in conventional fine disparity estimation with a small matching window size, but also can reduce the blocking effect, which occurs in coarse disparity estimation with a large matching window size. Some experimental results show that the proposed algorithm improves the peak signal-to-noise of the reconstructed intermediate view up to 2.93 to 4.09 dB and reduces the execution time to 39.34 to 65.62% on average in comparison with those of conventional algorithms.

Acquisition of sharp depth map from multiple cameras

We present a method to estimate a dense and sharp depth map using multiple cameras. A key issue for obtaining sharp depth map is how to overcome the harmful influence of occlusion. Thus, we first propose an occlusion-overcoming strategy which selectively uses the depth information from multiple cameras. With a simple selection technique, we resolve the occlusion problem considerably at a slight sacrifice of noise tolerance. Another key issue in area-based stereo matching is the size of matching window. We propose to use a hierarchical estimation scheme that attempts to acquire a sharp depth map such that edges of the depth map coincide with object boundaries on the one hand, and reduce noisy estimates due to insufficient size of matching window on the other hand. Owing to the unique property of occlusion-overcoming strategy, we can utilize full benefit of hierarchical schemes. We show the method can produce a sharp and correct depth map for a variety of images.