Pixel Matching Research Articles

Real-time registration of remote sensing images with a Markov chain model

The automatic registration of different remote sensing images of the same scene is an important problem in the processing of remote sensing images. Such images usually have different resolutions and computing an accurate registration of them in real time is thus often a challenging task. In this paper, a new statistical approach is developed for the accurate and efficient registration of two remote sensing images with different resolutions. The proposed approach utilizes a statistical model to evaluate the probability for each possible mapping between the two images and computes the one with the maximum probability for registration. Similar to most of the state-of-the-art methods for remote sensing image registration, the proposed approach assumes the existence of an affine transformation between the images to be registered. The registration is performed in three stages. In the first stage, the pixels in each image are efficiently partitioned into two sets with the Otsu’s algorithm and four approximate equations for the parameters in the affine transformation are established. In the second stage, pixels in both images with significant edge geometric features are selected for potential pixel matching. In the third stage, based on the four approximate equations generated in the first stage, an Markov Chain Model-based method is used to efficiently compute the matching that can pair the selected pixels with the maximum likelihood. The parameters of the affine transformation can be determined from the pixel pairs with a least square regression approach. Experimental results on a number of pairs of remote sensing images show that this approach can generate registration results more accurate than those obtained with a few state-of-the-art approaches. In addition, real-time evaluation also shows that the approach is computationally efficient and can be used in real-time applications for remote sensing image processing.

Local Stereo Matching: An Adaptive Weighted Guided Image Filtering-Based Approach

Innovative applications in rapidly evolving domains such as robotic navigation and autonomous (driverless) vehicles rely on binocular computer vision systems that meet stringent response time and accuracy requirements. A key problem in these vision systems is stereo matching, which involves matching pixels from two input images in order to construct the output, a 3D map. Building upon the existing local stereo matching algorithms, this paper proposes a novel stereo matching algorithm that is based on a weighted guided filtering foundation. The proposed algorithm consists of three main steps; each step is designed with the goal of improving accuracy. First, the matching costs are computed using a unique combination of complementary methods (absolute difference, Census, and gradient algorithms) to reduce errors. Second, the costs are aggregated using an adaptive weighted guided image filtering method. Here, the regularization parameters are adjusted adaptively using the Canny method, further reducing errors. Third, a disparity map is generated using the winner-take-all strategy; this map is subsequently refined using a densification method to reduce errors. Our experimental results indicate that the proposed algorithm provides a higher level of accuracy in comparison to a collection of the existing state-of-the-art local algorithms.

Pixel matching search algorithm for counting moving vehicle in highway traffic videos

Traffic monitoring through video processing is one of the hot research areas in the Intelligent Transportation System (ITS). Vehicle counting systems should be simple enough to be applied in real-time circumstances. A novel and fast algorithm for vehicle counting from a traffic video sequence is proposed in this paper where the vehicle tracking step is not necessary. A reference model is only created in the video frames for a narrow area. When going through this narrow area, the moving vehicles are identified as foreground objects. Detection of moving vehicles is achieved by integrating approximated median filter based background subtraction with binary integral projection. The detected candidates are counted as a vehicle using a novel pixel matching search algorithm. The proposed algorithm does not rely on every video frame. It only requires every third frame for processing and thus increases the computation speed by three times compared to existing techniques. The proposed algorithm is tested and validated on a standard data set as well as a custom data set. Two parameters such as accuracy and processing time are used for the system evaluation where an overall accuracy of 96.84% is achieved. The processing time results show that the proposed system can perform in real-time with an average real-time processing speed of 93.92%.

Spectral characteristics of talc and mineral abundance mapping in the Jahazpur Belt of Rajasthan, India using AVIRIS-NG data

ABSTRACT Spaceborne data with Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) Short Wave Infrared (SWIR) bands have been used worldwide in discriminating different clay minerals. However, due to wide bandwidth of ASTER sensors, it is difficult to distinguish between pure and impure variety of talc mineral. The present study highlights the efficacy of airborne hyperspectral data Airborne Visible/Infrared Imaging Spectrometer – Next Generation (AVIRIS-NG) for spectral analysis in talc mineral in the Jahazpur Belt of Bhilwara district in Rajasthan, India. Due to its better spectral and spatial resolution, AVIRIS-NG is found to be extremely useful in identifying talc mineral in SWIR region. Talc is characterized by doublet shaped spectral pattern in AVIRIS-NG image with strong absorption dips at 2289.75 nm and 2314.80 nm which could not be picked up in ASTER image. Spectral matching of talc pixels is verified with the field spectra of samples and available spectral library of United States Geological Survey, Jet Propulsion Laboratory, and International Geological Correlation Programme. Variation in reflectance percentage of different talc pixels is attributed to variation in chemical composition of talc. However, a large number of mixed pixels with spectral pattern of both talc and kaolinite minerals have been identified in AVIRIS-NG image which are found to be an impure variety of talc. Whole-pixel to sub-pixel classification techniques with Spectral Angle Mapper, Spectral Feature Fitting, Mixture Tuned Matched Filtering, and Linear Spectral Unmixing (LSU) techniques have been applied for talc abundance mapping in Jahazpur area. LSU is found to be the most useful and accurate technique in classifying spectrally pure to impure variety of talc.

Geometry and context guided refinement for stereo matching

The disparity refinement phase of existing end-to-end stereo matching networks refines the disparity by learning the mapping from the concatenated coarse disparity and corresponding features to fine disparity. It depends on the scenarios' characteristics, such as the distribution of disparity and semantic categories contained in the domain, which makes the network fail to work on unseen domain. In this paper, we propose a geometry and context guided refinement network (GCGR-Net) containing a Fine Matching module and an Upsampling module. GCGR-Net learns to utilize pixels' relationship to get high resolution dense disparity, which is independent of the data's content. The Fine Matching module performs a minimum range search based on the relationship between the possible matching pixel pairs, i.e. the called geometry information, to recover the internal structure of the object. The Upsampling module obtains context information, the relationship between central pixel and the pixels in its neighborhood, to upsample the lower resolution disparity. The final disparity map is obtained step by step through an iterative refinement model. Experiment results show that our method not only has good performance in the training scenarios, but also outperforms previous methods on the unseen domain without fine-tuning.

Three-Dimensional Measurement for Rigid Moving Objects Based on Multi-Fringe Projection

Multi-fringe projection profilometry, such as the widely used phase-shifting and temporal phase unwrapping algorithms, can achieve high accuracy and robustness in three-dimensional shape reconstruction of static objects. However, the motion of the objects will introduce errors. This paper proposes a new method to reconstruct 3D shape of rigid moving objects on an automated transmission line. First, a new pixel matching method for projected and captured images is proposed. Then, the pixel deviation caused by the motion of the objects in the three-dimensional direction is pre-detected, and the new projection fringes are obtained according to the above pixel matching. Third, the new projection fringes are binarized and the projector is defocused to realize high-speed projection. At last, the captured fringes are offset to reconstruct 3D shape. Experimental results show that the proposed method is effective in 3D shape reconstruction of rigid moving objects.

First study of Sentinel-3 SLSTR active fire detection and FRP retrieval: Night-time algorithm enhancements and global intercomparison to MODIS and VIIRS AF products

The Sea and Land Surface Temperature Radiometer (SLSTR) now operates concurrently onboard the European Sentinel-3A and 3B satellites. Its observations are expected ultimately to become the main global source of active fire (AF) detections and fire radiative power (FRP) retrievals for the mid-morning and evening low earth orbit timeslots – data currently supplied by the Moderate Resolution Imaging Spectroradiometer (MODIS) onboard Terra. Here we report for the first-time the significant adjustments made to the pre-launch Sentinel-3 AF detection and fire characterisation algorithm required to optimize its performance with real SLSTR data collected from the Sentinel-3A and 3B satellites. SLSTR possesses both an S7 ‘standard’ and an F1 ‘fire’ channel that operate in the same middle infrared (MIR) waveband, but which use different detectors with differing dynamic ranges and which are located at different focal plane locations. When S7 provides saturated observations, for example over higher FRP active fire pixels, F1 must be used to provide a reliable MIR spectral measurement. However, the two channels differing data characteristics (slightly different size, shape and spatial location of the matching pixels) means that swapping between their measurements is non-trivial. The main algorithm enhancement has therefore been the addition of a dedicated active fire pixel clustering component, required to cluster the detected AF pixels into individual fires as a solution to this issue. Focusing on night-time data due to the added complexity of daytime implementation, we compare AF information derived with this updated SLSTR algorithm to that from near-simultaneous MODIS Terra, and we find that SLSTR has a lower minimum FRP detection limit which enables more lower FRP active fire pixels to be identified than is the case with MODIS. When both sensors detect the same fire cluster at the same time, SLSTR typically measures a slightly higher FRP due to it being able to detect more of the low FRP AF pixels lying at the cluster edge (the OLS linear best fit between matched SLSTR and MODIS per-fire FRP matchups has a slope of 1.08). At the regional scale, SLSTR detects 90% of the AF pixels that the matching MODIS data contains, but also identifies an additional 44% more AF pixels – the vast majority of which have FRP < 5 MW. Regional FRP totals derived from SLSTR appear slightly higher than those from MODIS because of this, and the OLS linear best fit between these regional FRP matchup datasets has a slope of 1.10. Global fire mapping at 1° grid cell resolution for January 2019 shows very similar fire patterns and FRP totals from SLSTR onboard of Sentinel-3B and MODIS Terra, with SLSTR detecting seven times more AF pixels but very similar FRP totals. Case studies in 5° grid cell areas show the same pattern, and longer-term comparisons like these will provide the data required to mesh MODIS and SLSTR data into a single compatible time-series for long-term trend analysis. The night-time SLSTR AF product based largely on this algorithm has been fully operational from March 2020 and is available from near real-time feeds. A non-time critical (NTC) version based on a similar processing chain will follow shortly after, with products available from the Sentinel-3 Data Hub.

3-D Measurement Method for Nonuniform Reflective Objects

For 3-D measurement of nonuniform reflective objects, it is necessary to ensure measurement accuracy, reduce the number of additional images, and improve measurement efficiency. To satisfy these requirements, we first present a pixel matching method for projected and captured images. Second, we obtain the optimal projected image from two additional images. Finally, we propose a method to estimate reasonable exposure time. Experimental results show that the method can reconstruct 3-D shape of the metal, plastic, and colored objects with minimum of root mean square error (RMSE) 0.095, 0.068, and 0.079, respectively.

Fast 3D measurement based on improved optical flow for dynamic objects.

High resolution, real-time three-dimensional (3D) measurement plays an important role in many fields. In this paper, a multi-directional dynamic real-time phase measurement profilometry based on improved optical flow is proposed. In a five-step phase shifting dynamic measurement, pixel matching is needed to make the pixels one-to-one corresponding in five patterns. However, in the frequently-used pixel matching method at present, it is necessary to calculate the correlation and traverse the whole deformed pattern for the motion information of the measured object. The huge amount of computation caused by correlation computation takes up most of the time in the process of the entire 3D reconstruction, so it can not meet the requirement of real-time dynamic measurement. In order to solve the problem, the improved optical flow algorithm is introduced to replace correlation calculation in pixel matching. In one measurement, five captured patterns need to be dealt with, and the optical flow between each two adjacent frames is calculated. Then four two-dimensional vector matrices can be obtained. The vector matrices contain the complete motion information of the measured object. Experiments and simulations prove that this method can improve the efficiency of pixel matching by 42 times and 3D reconstruction by 32 times on the premise of ensuring the accuracy.

A new disparity map quality assessment based on structural similarity for remotely sensed image pairs

ABSTRACT Disparity map quality assessment is crucial to evaluate the accuracies of stereo matching algorithms. Several widely used measures such as root mean square (RMS), mean absolute error (MAE) and bad matching pixels (BMP) have been proposed to evaluate the similarity between disparity map and ground truth (GT) map. These measures are based on grey-scale errors while ignoring the structural similarity between disparity map and GT. In this paper, we propose a united similarity measure based on sructure similarity and normalized RMS (USNR) for disparity map quality assessment. Our new measure is able to simultaneously evaluate grey-scale similarity and region-aware structural similarity between disparity map and GT map. Experiments have been carried out using stereo pair datasets from various scenes, including street view, planetary, natural landscape and urban, to demonstrate the feasibility, reliability and robustness of our measure compared to state-of-the-art evaluation methods.

Improving Census Transform by High-Pass with Haar Wavelet Transform and Edge Detection.

One of the common methods for measuring distance is to use a camera and image processing algorithm, such as an eye and brain. Mechanical stereo vision uses two cameras to shoot the same object and analyzes the disparity of the stereo vision. One of the most robust methods to calculate disparity is the well-known census transform, which has the problem of conversion window selection. In this paper, three methods are proposed to improve the performance of the census transform. The first one uses a low-pass band of the wavelet to reduce the computation loading and a high-pass band of the wavelet to modify the disparity. The main idea of the second method is the adaptive size selection of the conversion window by edge information. The third proposed method is to apply the adaptive window size to the previous sparse census transform. In the experiments, two indexes, percentage of bad matching pixels (PoBMP) and root mean squared (RMS), are used to evaluate the performance with the known ground truth data. According to the results, the computation required can be reduced by the multiresolution feature of the wavelet transform. The accuracy is also improved with the modified disparity processing. Compared with previous methods, the number of operation points is reduced by the proposed adaptive window size method.

Feature matching based on curve descriptor and local D-Nets for fish-eye images.

Most feature-matching algorithms based on perspective images, such as scale-invariant feature transform (SIFT), speeded up robust features, or DAISY, construct their feature descriptors from the neighborhood information of feature points. Large nonlinear distortion results in different amounts of neighborhood information at different feature points within the fish-eye images, especially for the case when a feature pixel is at the central region and the corresponding feature pixel is at the periphery. In contrast, descriptor-Nets (D-Nets) is a feature-matching algorithm based on global information. It is more robust, but it is time-consuming. In this paper, we employ the SIFT detector to extract feature pixels, and then we propose a novel feature-matching strategy based on the D-Nets algorithm. We modify the linear descriptors in the traditional D-Nets algorithm and propose a curve descriptor based on the hemispheric model of a fish-eye image. In the traditional D-Nets algorithm, each feature point is described by all other pixels of the entire image, and complicated calculations cause slow matching speed. To solve this problem, we convert the traditional global D-Nets into a novel local D-Nets. In the experiment, we obtain image pairs from real scenery using the binocular fish-eye camera platform. Experimental results show that the proposed local D-Nets method can achieve more than 3 times the initial matching pixels, and the percentage of bad matching is reduced by 40% compared with the best performing method among the comparison methods. In addition, the matching pixel pairs obtained by the proposed method are evenly distributed, either in the center region with small distortion or in the peripheral region with large distortion. Meanwhile, the local D-Nets algorithm is 16 times less than that of the global D-Nets algorithm.

Controllable digital restoration of ancient paintings using convolutional neural network and nearest neighbor

Ancient paintings are valuable culture legacy which can help archaeologists and culture researchers to study history and humanity. Most ancient artworks have damage problems, such as degradation, flaking and cracking. This work presents a novel controllable image inpainting framework with capability of incorporating suggestions from experts, which can help artists envisage how the ancient painting may have looked after a restoration. The framework leverages the content prediction power of deep convolutional neural network (CNN) and the nearest neighbor based pixel matching, where a deep CNN is designed to produce a coarse estimation of complete paintings by filling in missing regions and nearest neighbor based pixel matching is designed to map a mid-frequency estimation obtained from the deep CNN to high quality outputs in a controllable manner. In addition, we design a pixel descriptor using multi-scale neural features from different layers of a pre-trained deep network to capture different amounts of spatial context. Experimental results demonstrate that the proposed approach successfully predicts information in large missing regions and generates controllable high-frequency photo-realistic inpainting results.

A Novel Stereo Matching Algorithm for Digital Surface Model (DSM) Generation in Water Areas

Image dense matching has become one of the widely used means for DSM generation due to its good performance in both accuracy and efficiency. However, for water areas, the most common ground object, accurate disparity estimation is always a challenge to excellent image dense matching methods, as represented by semi-global matching (SGM), due to the poor texture. For this reason, a great deal of manual editing is always inevitable before practical applications. The main reason for this is the lack of uniqueness of matching primitives, with fixed size and shape, used by those methods. In this paper, we propose a novel DSM generation method, namely semi-global and block matching (SGBM), to achieve accurate disparity and height estimation in water areas by adaptive block matching instead of pixel matching. First, the water blocks are extracted by seed point growth, and an adaptive block matching strategy considering geometrical deformations, called end-block matching (EBM), is adopted to achieve accurate disparity estimation. Then, the disparity of all other pixels beyond these water blocks is obtained by SGM. Last, the median value of height of all pixels within the same block is selected as the final height for this block after forward intersection. Experiments are conducted on ZiYuan-3 (ZY-3) stereo images, and the results show that DSM generated by our method in water areas has high accuracy and visual quality.

A novel augmented reality framework based on monocular semi‐dense simultaneous localization and mapping

SummaryMarkerless tracking has been a trend in augmented reality (AR) applications nowadays, but it no longer satisfies users who want virtual characters to interact with the real world such as collision. Some sparse or dense simultaneous localization and mapping (SLAM) methods are proposed aiming to solve this problem. However, sparse methods only extract a plane from the sparse map, which cannot allow virtual characters to move realistically. Meanwhile, dense methods usually require powerful graphics processing unit (GPU) for dense mapping. In this paper, we present a real‐time AR framework based on a semi‐dense method with central processing unit (CPU). Specifically, the semi‐dense method searches pixels with high gradients in each keyframe and estimates accurate depths by fusing matching pixels in other keyframes. We propose an outlier removal method that excludes three‐dimensional points outside the camera trajectory. By integrating this method, our framework preserves clean edges of the real environment. The experimental results on the dataset show that our proposed framework has better surface reconstruction accuracy than other methods and our tracking thread runs in an acceptable speed when the semi‐dense mapping thread runs backend. With the benefit of the robust camera tracking and the aligned surface, virtual characters of our AR application enable realistic movement and collision.

Automatic 3D color shape measurement system based on a stereo camera.

This study proposes an automatic three-dimensional (3D) color shape measurement system based on images recorded by a stereo camera. The system, comprising several off-the-shelf components, is cost-effective yet capable of obtaining quality color 3D objects. In the proposed system, a turntable carrying a checkerboard is used to assist the simultaneous calibration of the stereo camera and the turntable. A slit laser is driven to swing forward and backward for generating stripes on test objects. The stereo images are collected and analyzed for obtaining matching pixels, and, consequently, the 3D point coordinates based on epipolar geometry are obtained. Screened Poisson reconstruction is utilized to integrate and smooth the scanned surfaces. With additional color images from the same camera, several multi-view texturing methods are benchmarked. We concluded that our proposed system can successfully and automatically reconstruct quality 3D color shapes of various objects.

Dynamic phase measuring profilometry for rigid objects based on simulated annealing.

This paper presents a dynamic phase measurement profilometry (PMP) method based on the simulated annealing algorithm. In dynamic PMP for rigid objects, pixel matching is an effective method to make one-to-one pixel correspondence in each captured pattern. However, pixel matching by the global traversing algorithm takes up most of the time in the whole reconstruction process. For the purpose of optimizing pixel matching and enhancing performance in dynamic PMP, the simulated annealing algorithm is introduced. By generating a random path based on the simulated annealing algorithm, it is sufficient to locate the approximate area of the measured object. Then the accurate position can be calculated by combining it with a partial traversing algorithm. The proposed method can reduce pixel matching time by 63% and increase reconstruction efficiency by 58%. Simulations and experiments prove feasibility and precision.

A novel enhanced intensity-based automatic registration: Augmented reality for visualization and localization cancer tumors.

The purpose of this study is to replace the manual process (selecting the landmarks on mesh and anchor points on the video) by Intensity-based Automatic Registration method to reach registration accuracy and low processing time. The proposed system consists of an Enhanced Intensity-based Automatic Registration (EIbAR) using Modified Zero Normalized Cross Correlation (MZNCC) algorithm. The proposed system was implemented on videos of breast cancer tumors. Results showed that the proposed algorithm-as compared to a reference-improved registration accuracy by an average of 2 mm. In addition, the proposed algorithm-as compared to a reference-reduced the number of pixel matching, thereby reducing processing time on the video by an average of 22 ms/frame. The proposed system can, thus, provide an acceptable accuracy and processing time during scene augmentation of videos, which provides a seamless use of augmented-reality for surgeons in visualizing cancer tumors.

Medical Image Coloring Based on Gabor Filtering for Internet of Medical Things

Color medical images better reflect a patient's lesion information and facilitate communication between doctors and patients. The combination of medical image processing and the Internet has been widely used for clinical medicine on Internet of medical things. The classical Welsh method uses matching pixels to achieve color migration of grayscale images, but it exists problems such as unclear boundary and single coloring effect. Therefore, the key information of medical images after coloring can't be reflected efficiently. To address this issue, we propose an image coloring method based on Gabor filtering combined with Welsh coloring and apply it to medical grayscale images. By using Gabor filtering, which is similar to the visual stimulus response of simple cells in the human visual system, filtering in 4 directions and 6 scales is used to stratify the grayscale images and extract local spatial and frequency domain information. In addition, the Welsh coloring method is used to render the image with obvious textural features in the layered image. Our experiments show that the color transboundary problem can be solved effectively after the layered processing. Compared to images without stratification, the coloring results of the processed images are closer to the real image.

Disparity of Stereo Images by Self-Adaptive Algorithm

This paper introduces a new searching method named “Self Adaptive Algorithm (SAA)” for computing stereo correspondence or disparity of stereo image. The key idea of this method relies on the previous search result which increases searching speed by reducing the search zone and by avoiding false matching. According to the proposed method, stereo matching search range can be selected dynamically until finding the best match. The searching range -dmax to +dmax is divided into two searching regions. First one is -dmax to 0 and second one is 0 to +dmax .To determine the correspondence of a pixel of the reference image (left image), the window costs of the right image are computed either for -dmax to 0 region or for 0 to +dmax region depending only on the matching pixel position. The region where the window costs will be computed- will be automatically selected by the proposed algorithm based on previous matching record. Thus the searching range is reduced to 50% within every iteration. The algorithm is able to infer the upcoming candidate’s pixel position depending on the intensity value of reference pixel. So the proposed approach improves window costs calculation by avoiding false matching in the right image and reduces the search range as well. The proposed method has been compared with the state-of-the-art methods which were evaluated on Middlebury standard stereo data set and our SAA outperforms the latest methods both in terms of speed and gain enhancement with no degradation of accuracy.