Pixel Matching Research Articles

Quantification of the Uncertainty Caused by Geometric Registration Errors in Multiscale Validation of Satellite Products

Uncertainty quantification is an important part of validation, because the pixel scale reference generally suffers from uncertainty caused by different factors, lowering the accuracy of validation results. In order to take a step forward to characterize the uncertainty of validation results, this study proposed a simulated shift-based pixel matching (SSPM) method with the aim of quantifying the uncertainty caused by geometric mismatch in the multiscale validation. Furthermore, its relationships with spatial heterogeneity and subpixel size were also explored. It was found that the uncertainty caused by the geometric mismatch is nonnegligible in multiscale validation, which would obscure the true accuracy of satellite products. Spatial heterogeneity makes a positive contribution to the uncertainty caused by geometric mismatch, but the magnitude depends on subpixel size, being weaker with small subpixel size and stronger with larger subpixel size. Subpixel size is generally positively related to geometric uncertainty. But in the case of very large spatial heterogeneity, their correlation is very weak. This study is an important step toward quantitatively characterizing the uncertainties of pixel scale reference in order to increase the confidence of validation results.

Deep View Synthesis via Self-Consistent Generative Network

View synthesis aims to produce unseen views from a set of views captured by two or more cameras at different positions. This task is non-trivial since it is hard to conduct pixel-level matching among different views. To address this issue, most existing methods seek to exploit the geometric information to match pixels. However, when the distinct cameras have a large baseline (i.e., far away from each other), severe geometry distortion issues would occur and the geometric information may fail to provide useful guidance, resulting in very blurry synthesized images. To address the above issues, in this paper, we propose a novel deep generative model, called Self-Consistent Generative Network (SCGN), which synthesizes novel views from the given input views without explicitly exploiting the geometric information. The proposed SCGN model consists of two main components, i.e., a View Synthesis Network (VSN) and a View Decomposition Network (VDN), both employing an Encoder-Decoder structure. Here, the VDN seeks to reconstruct input views from the synthesized novel view to preserve the consistency of view synthesis. Thanks to VDN, SCGN is able to synthesize novel views without using any geometric rectification before encoding, making it easier for both training and applications. Finally, adversarial loss is introduced to improve the photo-realism of novel views. Both qualitative and quantitative comparisons against several state-of-the-art methods on two benchmark tasks demonstrated the superiority of our approach.

Prospects for Solar-Induced Chlorophyll Fluorescence Remote Sensing from the SIFIS Payload Onboard the TECIS-1 Satellite

The importance of solar-induced chlorophyll fluorescence (SIF) to monitoring vegetation photosynthesis has attracted much attention from the ecological and remote sensing research communities. Space-borne SIF products have been obtained owing to the rapid development of atmospheric satellites in recent years. The SIF Imaging Spectrometer (SIFIS) is a payload onboard the upcoming Terrestrial Ecosystem Carbon Inventory Satellite (TECIS-1) that is specifically designed for SIF monitoring. We conducted an in situ experiment to evaluate the performance of SIFIS on spectral measurement and SIF retrieval through comparison to the commercial spectrometer QE Pro. Disregarding the spatiotemporal mismatch between the collected measurements of the two spectrometers, the radiance spectra obtained synchronously by SIFIS and QE Pro showed a high level of consistency. The SIF retrieval, normalized difference vegetation index (NDVI), and near-infrared radiance of vegetation (NIRvR) results for a push-broom image shows consistent spatial distributions over both vegetated and nonvegetated surfaces. A quantitative comparison was conducted by strictly filtering matching pixels. For the far-red band, a high correlation was obtained between the SIF retrieval performances of SIFIS and QE Pro with R 2 = 0.70 and RMSE = 0.30 mW m − 2 s r − − 1 n m − 1 . However, a relatively poor correlation was observed for the red band with an R 2 value of 0.23 and an RMSE of 0.26 mWm −2 sr -−1 nm −1 . Despite the large uncertainties associated with this experiment, the results indicate that TECIS-1 should offer a reliable SIF monitoring performance after its launch.

A new method for watermarking color images using virtual hiding and El-Gamal ciphering

One of the important issues in the era of computer networks and multimedia technology development is to find ways to maintain the reliability, credibility, copyright and non-duplication of digital content transmitted over the internet. For the purpose of protecting images from illegal usage, a watermark is used. A hidden digital watermark is the process of concealing information on a host to prove that this image is owned by a specific person or organization. In this paper, a new method has been proposed to use an RGB logo to protect color images from unlicensed trading. The method depends on retrieving logo data from specific locations in the host to form a logo when the owner claims the rights to those images. These positions are chosen because their pixels match the logo data. The locations of matching pixels are stored in a table that goes through two stages of treatment to ensure confidentiality: First, table compression, second, encoding positions in the compressed table through El-Gamal algorithm. Because the method depends on the idea of keeping host pixels without change, PSNR will always be infinity. After subjecting the host to five types of attack, the results demonstrate that the method can effectively protect the image and hidden logo is retrieved clearly even after the attacks.

High-precision 3D shape measurement of rigid moving objects based on the Hilbert transform.

Phase-shifting profilometry (PSP) is a three-dimensional (3D) measurement method of point-to-point calculation. The consistency of object position is the prerequisite to ensure the successful application of PSP in moving objects. The position information of an object can be well characterized by the modulation patterns, and hence a high-quality modulation pattern is the guarantee of pixel-matching accuracy. In this paper, a generic modulation pattern enhancement method for rigid moving objects based on the Hilbert transform is proposed. First, the Hilbert transform is employed to suppress the zero-frequency components of the fringe pattern, and a hybrid digital filter window is applied to filter out the positive fundamental frequency components for a higher signal-to-noise ratio. Then the grid-based motion statistics for fast, ultra-robust feature correspondence algorithm is used to match the high-quality modulation patterns between two adjacent frames, and the object positions in the three deformed patterns are made consistent by image clipping. Finally, the three-step PSP is used to reconstruct the 3D shape of the measured object. Experimental results demonstrate that the proposed method can substantially improve the quality of the modulation pattern, achieve high-precision pixel matching, and ultimately reduce the motion-introduced phase error.

Histogram-derived modified thresholds for coronary artery calcium scoring with lower tube voltage

We aimed to determine the proper modified thresholds for detecting and weighting CAC scores at 100 kV through histogram matching in comparison with 120 kV as a standard reference. From the training set (680 participants), modified thresholds at 100 kV were obtained through histogram matching of calcium pixels to 120 kV. From the validation set (213 participants), a standard CAC score at 120 kV, and modified CAC score at 100 kV using modified thresholds were compare through the paired t test and the Bland–Altman plot. Agreement for risk categories (no, minimal, mild, moderate, and severe) was evaluated using kappa statistics. Radiation doses were also compared. For the validation set, there was no significant difference between standard (median, 18.7; IQR, 0.0–207.0) and modified (median, 17.3; IQR, 0.0–220.9) CAC scores (P = 0.689). A small bias was achieved (0.74) with 95% limits of agreement from − 52.35 to 53.83. Agreements for risk categories were excellent (κ = 0.994). The mean dose-length-product of 100-kV scanning (30.1 ± 0.8 mGy * cm) was significantly decreased compared to 120-kV scanning (42.9 ± 0.6 mGy * cm) (P < 0.001). Histogram-derived modified thresholds at 100 kV can enable accurate CAC scoring while reducing radiation exposure.

Effective and efficient pixel-level detection for diverse video copy-move forgery types

Video copy-move forgery detection (VCMFD) is a significant and greatly challenging task due to a variety of difficulties, including a huge amount of video information, diverse forgery types, rich forgery objects, and homogenous forgery sources. These difficulties raise four unresolved key challenges in VCMFD: i) ineffective detection in some popular forgery cases; ii) inefficient matching in processing numerous video pixels with hundred-dimensional features under dozens of matching iterations; iii) high false positive (FP) in detecting forgery videos; iv) low trade-off of efficiency and effectiveness in filling forgery region, and even failing in indicating forgeries at the pixel level. In this paper, a novel VCMFD method is proposed to address these issues: i) an innovatively improved SIFT structure that can address the thorough feature extraction in all video copy-move forgery cases; ii) a novel fast keypoint-label matching (FKLM) algorithm is proposed that creates some keypoint-label groups so that every high-dimensional feature is assigned into one of these groups. As a result, matching of video pixels can be directly done on a small number of keypoint-label groups only, leading to a nearly 500% raise in matching efficiency; iii) a new coarse-to-fine filtering relying on intrinsic attributes of exact keypoint-matches is designed to more effectively reduce the false keypoint-matches; iv) the adaptive block filling relying on true keypoint-matches contributes to the accurate and efficient suspicious region filling, even at the pixel level. Finally, the suspicious region locations with the forgery vision persistence concept indicate forgery videos. Compared to the state-of-art methods, the experiments show that our proposed method achieves the best detection accuracy, lowest FP, and improved at least 16% and 8% of F1 scores on the GRIP 2.0 dataset and a combination of SULFA 2.0 & REWIND datasets. Furthermore, the proposed method is with low computational time (4.45 s/Mpixels), which is about 1/2-1/3 times of the latest DFMI-BM (8.02 s/Mpixels) and PM-2D (13.1 s/Mpixels) methods.

3D Surface reconstruction of transparent objects using laser scanning with LTFtF method

Three-dimensional (3D) vision plays an important role in robotics and industrial automation, wherein the reconstruction of transparent objects poses a great challenge. In this paper, we present a scheme for reconstructing the 3D surfaces of transparent objects based on a combination of stereo matching and laser scanning. The traditional laser scanning method can achieve good results in the 3D reconstruction of opaque objects. However, this method will fail for transparent and semitransparent objects because both the front and back surfaces of these objects will reflect the feature laser line. It is difficult to determine the reflection surface of the feature lines captured by the camera. To distinguish the laser reflected by the front surface from the distracting laser reflections by the other surfaces, a laser tracking frame to frame (LTFtF) method is proposed. The laser lines reflected from the front surface of the target are then reconstructed using stereo vision. The entire 3D surface is obtained through laser scanning. In addition, we improve the matching accuracy and point cloud density through an internal pixel matching method. Various experiments were conducted to verify the capability of the proposed method, which includes the reconstruction of transparent and semitransparent objects of various shapes. The proposed method was compared with traditional laser scanning algorithms.

Fast and robust online three-dimensional measurement based on feature correspondence

Online three-dimensional (3D) measurement plays an important role in industry. When phase-shifting profilometry is employed in online 3D measurement, pixel matching is an important step to keep objects at the same coordinate value. However, the correlation operation and marker feature matching algorithms may take a long time, increasing the complexity. So a fast and robust online 3D measurement based on feature correspondence is proposed. In this method, only one frame of the sinusoidal fringe pattern is projected onto the measured object, and image correction technique is employed to rectify pixel size. Then five frames of deformed patterns with equivalent displacement are captured by the camera, and the corresponding modulation patterns are extracted. The oriented fast and rotated brief feature algorithm is used to extract the matching pair of feature points, and the improved grid-based motion statistical feature algorithm can better eliminate the false match to achieve pixel matching. In this way, five frames of deformed patterns with an equivalent shifted-phase can be extracted. Finally, the 3D shape of the measured object is reconstructed by the five-step equivalent phase-shifting algorithm. Experimental results verify the effectiveness and feasibility of the proposed method.

Geometric- and Optimization-Based Registration Methods for Long-Wave Infrared Hyperspectral Images

Registration of long-wave infrared (LWIR) hyperspectral images with their thermal and emissivity components has until now received comparatively less attention with respect to the visible near and short wave infrared hyperspectral images. In this paper, the registration of LWIR hyperspectral images is investigated to enhance applications of LWIR images such as change detection, temperature and emissivity separation, and target detection. The proposed approach first searches for the best features of hyperspectral image pixels for extraction and matching in the LWIR range and then performs a global registration over two-dimensional maps of three-dimensional hyperspectral cubes. The performances of temperature and emissivity features in the thermal domain along with the average energy and principal components of spectral radiance are investigated. The global registration performed over whole 2D maps is further improved by blockwise local refinements. Among the two proposed approaches, the geometric refinement seeks the best keypoint combination in the neighborhood of each block to estimate the transformation for that block. The alternative optimization-based refinement iteratively finds the best transformation by maximizing the similarity of the reference and transformed blocks. The possible blocking artifacts due to blockwise mapping are finally eliminated by pixelwise refinement. The experiments are evaluated with respect to the (i) utilized similarity metrics in the LWIR range between transformed and reference blocks, (ii) proposed geometric- and optimization-based methods, and (iii) image pairs captured on the same and different days. The better performance of the proposed approach compared to manual, GPU-IMU-based, and state-of-the-art image registration methods is verified.

The 3D Neural Network for Improving Radar-Rainfall Estimation in Monsoon Climate

The reflectivity (Z)—rain rate (R) model has not been tested on single polarization radar for estimating monsoon rainfall in Southeast Asia, despite its widespread use for estimating heterogeneous rainfall. The artificial neural network (ANN) regression has been applied to the radar reflectivity data to estimate monsoon rainfall using parametric Z-R models. The 10-min reflectivity data recorded in Kota Bahru radar station (in Malaysia) and hourly rain record in nearby 58 gauge stations during 2013–2015 were used. The three-dimensional nearest neighbor interpolation with altitude correction was applied for pixel matching. The non-linear Levenberg Marquardt (LM) regression, integrated with ANN regression minimized the spatiotemporal variability of the proposed Z-R model. Results showed an improvement in the statistical indicator, when LM and ANN overestimated (6.6%) and underestimated (4.4%), respectively, the mean total rainfall. For all rainfall categories, the ANN model has a positive efficiency ratio of &gt;0.2.

A discovery about the positional distribution pattern among candidate homologous pixels and its potential application in aerial multi-view image matching

In aerial multi-view photogrammetry, whether there is a special positional distribution pattern among candidate homologous pixels of a matching pixel in the multi-view images? If so, can this positional pattern be used to precisely confirm the real homologous pixels? These problems have not been studied at present. Therefore, the study of the positional distribution pattern among candidate homologous pixels based on the adjustment theory in surveying is investigated in this paper. Firstly, the definition and computing method of pixel’s pseudo object-space coordinates are given, which can transform the problem of multi-view matching for confirming real homologous pixels into the problem of surveying adjustment for computing the pseudo object-space coordinates of the matching pixel. Secondly, according to the surveying adjustment theory, the standardized residual of each candidate homologous pixel of the matching pixel is figured out, and the positional distribution pattern among these candidate pixels is theoretically inferred utilizing the quantitative index of standardized residual. Lastly, actual aerial images acquired by different sensors are used to carry out experimental verification of the theoretical inference. Experimental results prove not only that there is a specific positional distribution pattern among candidate homologous pixels, but also that this positional distribution pattern can be used to develop a new object-side multi-view image matching method. The proposed study has an important reference value on resolving the defects of existing image-side multi-view matching methods at the mechanism level.

Monocular Visual Odometry based on joint unsupervised learning of depth and optical flow with geometric constraints

Inferring camera ego-motion from consecutive images is essential in visual odometry (VO). In this work, we present a jointly unsupervised learning system for monocular VO, consisting of single-view depth, two-view optical flow, and camera-motion estimation module. Our work mitigates the scale drift issue which can further result in a degraded performance in the long-sequence scene. We achieve this by incorporating standard epipolar geometry into the framework. Specifically, we extract correspondences over predicted optical flow and then recover ego-motion. Additionally, we obtain pseudo-ground-truth depth via triangulating 2D-2D pixel matches, which makes the depth scale is closely relevant to the pose. Experimentation on the KITTI driving dataset shows competitive performance compared to established methods.

Refinement of matching costs for stereo disparities using recurrent neural networks

Depth is essential information for autonomous robotics applications that need environmental depth values. The depth could be acquired by finding the matching pixels between stereo image pairs. Depth information is an inference from a matching cost volume that is composed of the distances between the possible pixel points on the pre-aligned horizontal axis of stereo images. Most approaches use matching costs to identify matches between stereo images and obtain depth information. Recently, researchers have been using convolutional neural network-based solutions to handle this matching problem. In this paper, a novel method has been proposed for the refinement of matching costs by using recurrent neural networks. Our motivation is to enhance the depth values obtained from matching costs. For this purpose, to attain an enhanced disparity map by utilizing the sequential information of matching costs in the horizontal space, recurrent neural networks are used. Exploiting this sequential information, we aimed to determine the position of the correct matching point by using recurrent neural networks, as in the case of speech processing problems. We used existing stereo algorithms to obtain the initial matching costs and then improved the results by utilizing recurrent neural networks. The results are evaluated on the KITTI 2012 and KITTI 2015 datasets. The results show that the matching cost three-pixel error is decreased by an average of 14.5% in both datasets.

Three-dimensional (3D) reconstructions of the coastal cliff face in Normandy (France) based on oblique Pléiades imagery: assessment of Ames Stereo Pipeline® (ASP®) and MicMac® processing chains

ABSTRACT Images from agile (viewing angle over 40°) and very high spatial resolution satellites (inferior to 1 m) can be useful for monitoring cliff faces, which is the best proxy to better understand coastal cliff dynamics. However, these images with a specific configuration are rarely used, partly because it is cumbersome to process them. Based on Pléiades images of the coastal cliff face along the coast of Normandy, with a high angle of incidence (up to 40°) and taken on multiple dates, the paper aims to identify i) the best open-source processing chain to reconstitute three-dimensional (3D) cliff faces by stereo restitution ii) the reasons behind its best performance and iii) the key parameters to change depending on the image datasets or processing chains so as to facilitate transposition. The Ames Stereo Pipeline® (ASP®) and MicMac® software programmes were tested using different parameters (matching algorithm, size of correlation window, etc.) for the 3D reconstructions. MicMac® provides the best performance using GeomImage (1–2 pixel matching) with a size of correlation window of 3 × 3 or 7 × 7 associated with a regularization parameter of 0.10. With these parameters, the point clouds of the cliff face have an average point density of 1.70 point m−2, a mean distance from Unmanned Aerial Vehicle (UAV) ground truth data of 0.04 m and a standard deviation of 1.72 m. With these characteristics, the threshold of rockfall detection using a multi-source comparison is assessed at 100 m3, which involves that the large majority of rockfalls (69%) around the study area could be detected by a diachronic approach. Considering the daily Pléiades revisiting time, this method offers a great opportunity to monitor erosion and to better understand coastal cliff dynamics.

A novel entropy-based texture inpainting algorithm

Image inpainting is the process of restoring a lost or damaged portion of an image. Inpainting of an image that contains texture remains a particularly challenging problem. We aim to propose an algorithm to inpaint a textured image accurately using a single image. The main idea is to segment the given image, based on its texture. In this work, we propose a novel local energy approach, in combination with the k-means algorithm to segment the given image, based on its texture. We use this segmentation result to restrict the search of matching pixels to only-relevant segments. Moreover, we use the entropy-based dissimilarity parameter to find matching pixels, instead of the $$\ell ^2$$ distance. The restriction of the search area improves the efficiency, and the use of the proposed dissimilarity parameter provides a better way to compare textures, giving improved inpainting for textured images.

On the Confidence of Stereo Matching in a Deep-Learning Era: A Quantitative Evaluation.

Stereo matching is one of the most popular techniques to estimate dense depth maps by finding the disparity between matching pixels on two, synchronized and rectified images. Alongside with the development of more accurate algorithms, the research community focused on finding good strategies to estimate the reliability, i.e., the confidence, of estimated disparity maps. This information proves to be a powerful cue to naively find wrong matches as well as to improve the overall effectiveness of a variety of stereo algorithms according to different strategies. In this paper, we review more than ten years of developments in the field of confidence estimation for stereo matching. We extensively discuss and evaluate existing confidence measures and their variants, from hand-crafted ones to the most recent, state-of-the-art learning based methods. We study the different behaviors of each measure when applied to a pool of different stereo algorithms and, for the first time in literature, when paired with a state-of-the-art deep stereo network. Our experiments, carried out on five different standard datasets, provide a comprehensive overview of the field, highlighting in particular both strengths and limitations of learning-based strategies.

A Dense 3-D Point Cloud Measurement Based on 1-D Background-Normalized Fourier Transform

The 3-D surface measurement plays a significant role in the industrial area. With the advantages of high resolution and acquisition rate, dual-line-scan camera systems have been gradually studied for dynamic 3-D measurement. However, the partial overexposure and the limited measurement depth range of dual-line-scan cameras are the two main elements affecting the quality of the point cloud. Both of these problems are caused by incorrect pixel matching. Therefore, this article presents a matching strategy for dual-line-scan cameras based on the 1-D background-normalized Fourier transform (1DBNFT) to expand the depth range of measurement and deal with the partial overexposure. The wrapped phases extracted by 1DBNFT, the unwrapped phase extracted by projection distance minimization (PDM), and the matching principle are described in detail. We also analyze the possible errors in the system from three aspects: noncoplanar error, installation error, and motion error. Finally, the comparative experiments and accuracy verification experiments demonstrate the effectiveness of our algorithm.

Self-Guided Stereo Correspondence Estimation Algorithm

This paper introduces an innovative algorithm, “Self-guided Stereo Correspondence” (SGSC), that is directed by photometric properties of the candidate pixels. As the photometric properties of reference image (left image) pixel and its neighbor’s pixel are similar in most cases, so the upcoming corresponding pixel exists in the surrounding of the previous matching pixel. Searching performance is greatly improved by utilizing this photometric property of the candidate pixels. The searching performance is further improved by applying the pioneering threshold technique. These two key techniques sufficiently reduced the computational cost with no degradation of accuracy. The achievements of the proposed method are testified on Middlebury standard Stereo Datasets of 2003 and 2006 and the Middlebury latest Optical Flow Dataset. Finally, the proposed method is compared with present state-of-the-art methods and our SGSC outdoes the latest methods in terms of speed, visualization of hidden ground truth, 3D reconstruction and accuracy.

Road Surface Reconstruction by Stereo Vision

This paper covers the problem of road surface reconstruction by stereo vision with cameras placed behind the windshield of a moving vehicle. An algorithm was developed that employs a plane-sweep approach and uses semi-global matching for optimization. Different similarity measures were evaluated for the task of matching pixels, namely mutual information, background subtraction by bilateral filtering, and Census. The chosen sweeping direction is the plane normal of the mean road surface. Since the cameras’ position in relation to the base plane is continuously changing due to the suspension of the vehicle, the search for the base plane was integrated into the stereo algorithm. Experiments were conducted for different types of pavement and different lighting conditions. Results are presented for the target application of road surface reconstruction, and they show high correspondence to laser scan reference measurements. The method handles motion blur well, and elevation maps are reconstructed on a millimeter-scale, while images are captured at driving speed.