Zero-mean Normalized Cross Correlation Research Articles

Coded speckle target: Design, analysis and applications

In this paper, a novel encoding feature based on speckle patterns. It comprises three components: a pre-positioning ring, localization speckle area, and encoding speckle area. The pre-positioning ring identifies potential regions of interest. Shape function of the localization speckle area can be estimated by formulating the elliptical equation of the pre-positioning ring. Next, digital image correlation method is employed to achieve precise positioning and refine the shape function. Through the high-precision shape function, the encoded speckle area can be determined. Encoding technique utilizes a reference encoding scheme, in which each coded value region either same or is inverse to the reference region. Zero-mean normalized cross correlation relative to the reference region can be used to obtain a highly robust decoding value. Experimental results demonstrate that, despite challenges such as noise, defocus, variations in lighting intensity, and lighting uneven, the coded speckle target maintains high robustness and precise positioning accuracy. Moreover, the structure from motion and displacement measurement experiments further verifies its high accuracy in practical applications.

A Multimode Microfiber Specklegram Biosensor for Measurement of CEACAM5 through AI Diagnosis.

Carcinoembryonic antigen (CEACAM5), as a broad-spectrum tumor biomarker, plays a crucial role in analyzing the therapeutic efficacy and progression of cancer. Herein, we propose a novel biosensor based on specklegrams of tapered multimode fiber (MMF) and two-dimensional convolutional neural networks (2D-CNNs) for the detection of CEACAM5. The microfiber is modified with CEA antibodies to specifically recognize antigens. The biosensor utilizes the interference effect of tapered MMF to generate highly sensitive specklegrams in response to different CEACAM5 concentrations. A zero mean normalized cross-correlation (ZNCC) function is explored to calculate the image matching degree of the specklegrams. Profiting from the extremely high detection limit of the speckle sensor, variations in the specklegrams of antibody concentrations from 1 to 1000 ng/mL are measured in the experiment. The surface sensitivity of the biosensor is 0.0012 (ng/mL)-1 within a range of 1 to 50 ng/mL. Moreover, a 2D-CNN was introduced to solve the problem of nonlinear detection surface sensitivity variation in a large dynamic range, and in the search for image features to improve evaluation accuracy, achieving more accurate CEACAM5 monitoring, with a maximum detection error of 0.358%. The proposed fiber specklegram biosensing scheme is easy to implement and has great potential in analyzing the postoperative condition of patients.

3D Feature-Based Sampled-Data Visual Tracking*

3D feature-based Visual Servoing (VS) on the one hand shows attractive peculiarities, on the other hand it suffers from drawbacks related to the existence of local minima, which may affect the convergence character of the VS control loop. Furthermore, the performance of the visual tracking module may constitute a bottleneck enforcing severe constraints on the workspace and visual task execution speed. In this paper we introduce a novel sampled-data model of the 3D feature-based VS, and, in order to avoid drawbacks due to local minima, we plan the target reference trajectory in the feature space with the aim to constraint the feature error dynamics to remain close to the desired equilibrium point. Then, we propose a novel feature generation based on the homography provided by a template matching algorithm based on the Zero mean Normalized Cross Correlation (ZNCC) and the design of a visual tracking scheme by resorting to the Extended Kalman Filter (EKF) and Lyapunov direct method, which explicitly takes into account the camera velocity limits, while ensuring stability.

An Epipolar HS-NCC Flow Algorithm for DSM Generation Using GaoFen-3 Stereo SAR Images

Radargrammetry is a widely used methodology to generate the large-scale Digital Surface Model (DSM). Stereo matching is the most challenging step in radargrammetry due to the significant geometric differences and the inherent speckle noise. The speckle noise results in significant grayscale differences of the same feature points, which makes the traditional Horn–Schunck (HS) flow or multi-window zero-mean normalized cross-correlation (ZNCC) methods degrade. Therefore, this paper proposes an algorithm named Epipolar HS-NCC Flow (EHNF) for dense stereo matching, which is an improved HS flow method with normalized cross-correction constraint based on epipolar stereo images. First, the epipolar geometry is applied to resample the image to realize the coarse stereo matching. Subsequently, the EHNF method forms a global energy function to achieve fine stereo matching. The EHNF method constructs a local normalized cross-correlation constraint term to compensate for the grayscale invariance constraint, especially for the SAR stereo images. Additionally, two assessment methods are proposed to calculate the optimal cross-correlation parameter and smoothness parameter according to the refined matched point pairs. Two GaoFen-3 (GF-3) image pairs from ascending and descending orbits and the open Light Detection and Ranging (LiDAR) data are utilized to fully evaluate the proposed method. The results demonstrate that the EHNF algorithm improves the DSM elevation accuracy by 9.6% and 27.0% compared with the HS flow and multi-window ZNCC methods, respectively.

Fast template matching in multi-modal image under pixel distribution mapping

A scheme that can achieve fast template matching by both traditional methods and neural network methods, called Hybrid Matching by Pixel Distribution Mapping (HMPDM), is proposed for the matching difficulty caused by the problem of nonlinear intensity differences between multi-modal images. The HMPDM scheme first extracts the distribution information of image pixels through the developed expanded Slice Transform (eSLT) matrix, to overcome the nonlinear intensity differences between visible and thermal infrared images; and then maps the eSLT matrices corresponding to different modal images into correlation surface images with the same modality through traditional integer mapping mechanism or neural network mapping mechanism; finally, template matching between visible and thermal infrared images is achieved by evaluating the similarity of correlation surface images through the Zero-mean Normalized Cross-Correlation (ZNCC) algorithm. The fast and classic ZNCC algorithm ensures the real-time performance of the HMPDM scheme in the most time-consuming sliding window matching stage. The traditional integer mapping mechanism and the neural network mapping mechanism enable the HMPDM scheme to use the traditional method and the neural network method alone to achieve template matching, or to combine the two methods to achieve template matching. The comprehensive utilization of traditional methods and neural network methods makes the HMPDM scheme adaptable to various platforms with limited hardware resources. The experimental results show that, on a hardware platform with limited resources, it only takes 0.18 s for a 64 × 64 template image to slide on a 256 × 256 query image to achieve template matching; and it only takes at least 1.2 s for neural network training on the CPU. The matching performance is also better than many popular multi-modal image template matching algorithms.

Non-contact measurement method for reconstructing three-dimensional scour depth field based on binocular vision technology in laboratory

The scour of pile foundations significantly impacts the bridge structure safety. This study proposes a three-dimensional (3D) non-contact scour pit depth measurement method. Cameras were calibrated using a three-step calibration procedure based on the diagonal length. Refraction errors caused by the interface between water and air were corrected. The corresponding solutions are given to the problem that it is difficult to mark sand with traditional speckles, and the measurement range of a single camera is limited. The matching of virtual grid points between different views at different moments was achieved using the theory of digital image correlation (DIC) based on zero-mean normalized cross-correlation (ZNCC). The scour pit was reconstructed by splicing the depth field from different views. The comparison between the results of direct and visual measurements at a single point proves that the difference between them was basically at the millimeter level. Scour pits of different water depths, velocities, reciprocating flows, and unidirectional flows were photographed and reconstructed. Based on these results, the development of scour pits under various flow conditions is discussed.

A novel complementation method of an acoustic shadow region utilizing a convolutional neural network for ultrasound-guided therapy.

Noise-free ultrasound images are essential for organ monitoring during regional ultrasound-guided therapy. When the affected area is located under the ribs, however, acoustic shadow is caused by the reflection of sound from hard tissues such as bone, and the image is output with missing information in this region. Therefore, in the present study, we attempt to complement the image in the missing area. The overall flow of the complementation method to generate a shadow-free composite image is as follows. First, we constructed a binary classification method for the presence or absence of acoustic shadow on a phantom kidney based on a convolutional neural network. Second, we created a composite shadow-free image by searching for a suitable image from a time-series database and superimposing the corresponding area without shadow onto the missing area of the target image. In addition, we constructed and verified an automatic kidney mask generation method utilizing U-Net. The complementation accuracy for kidney tracking could be enhanced by template matching. Zero-mean normalized cross-correlation (ZNCC) values after complementation were higher than that of before complementation under four different data generation conditions: (i) changing the position of the bed of the robotic ultrasound diagnostic system in the translational direction, (ii) changing the probe angle in the translational direction, (iii) with the addition of rotational motion of the probe to condition (ii). Although there was large variation in the shape of the kidney contour in condition (iii), the proposed method improved the ZNCC value from 0.5437 to 0.5807. The effectiveness of the proposed method was demonstrated in phantom experiments. Verification of its effectiveness in real organs is necessary in future study.

Low Velocity Impact Localization of Variable Thickness Composite Laminates.

Variable thickness composite laminates (VTCL) are susceptible to impact during use and may result in irreparable internal damage. In order to locate the internal impact damage of complex composite structures and monitor the impact signals of VTCL at the same time, a low velocity impact (LVI) monitoring system based on an optical fiber sensing network was constructed. Fiber Bragg grating (FBG) sensors are suitable for monitoring strain characteristics. By arranging FBG sensors on the laminate, we studied the spectrum analysis and localization of the impact signal collected by a FBG demodulator at constant temperature. The prior knowledge of variable thickness composite structures is difficult to obtain, and the multi-sensor dynamic monitoring is complex and difficult to realize. In order to locate the LVI of composite structures without prior knowledge, based on empirical mode decomposition (EMD), we proposed an impact localization method with zero-mean normalized cross-correlation (ZNCC) and thickness correction. The experimental results of LVI localization verification show that the ZNCC algorithm can effectively remove the temperature cross-sensitivity and impact energy influencing factors, and the thickness correction can reduce the interference of variable thickness characteristics on localization performance. The maximum localization error is 24.41 mm and the average error is 15.67 mm, which meets engineering application requirements. The method of variable-thickness normalization significantly improves impact localization performance for VTCL.

Variable subset DIC algorithm for measuring discontinuous displacement based on pixel-level ZNCC value distribution map

Digital image correlation (DIC) is one of the most used techniques for measuring deformation. It is widely used in mechanical performance testing, structural health monitoring and many other fields. The standard subset based DIC technology assumes that the deformation is continuous. However, discontinuous deformation such as cracks and dislocations are quite common in practical applications. The measurement accuracy of DIC in these areas will be significantly reduced, or even measurement failure. This is a limitation for the application of DIC. To overcome this problem, this paper proposes a variable subset DIC algorithm. Taking the zero-mean normalized cross-correlation (ZNCC) function as the correlation criterion, the concept of pixel-level ZNCC value distribution map is proposed. The proposed algorithm can automatically select the most suitable subset for matching under the guidance of the pixel-level ZNCC value distribution map. This method has the advantages of good flexibility, strong applicability, and high precision.

Sub-pixel displacement measurement based on the combination of a gray wolf optimizer and gradient algorithm.

The digital speckle correlation method (DSCM) aims to measure the displacement of the interesting points by matching the subset around the same point between the undeformed image and the deformed image. It is an effective and powerful optical metrology method for deformation measurement. Considering that the gray wolf optimizer (GWO) is one of the most popular metaheuristic algorithms to calculate the unknown search spaces in the field of optical engineering, a sub-pixel displacement measurement technique based on the GWO and gradient algorithm is proposed. First, the zero-mean normalized cross correlation function is applied to analyze the correlation between the reference image and deformed image subsets. Second, by exploiting the global searching ability of the GWO algorithm, the initial integer pixel value is obtained and further viewed as the initialization displacement. Finally, the final sub-pixel displacement is generated by using a Barron gradient algorithm. Compared with the state-of-the-art methods on synthetic speckle images, the proposed method can effectively measure the displacement and deformation of rigid bodies. Furthermore, the experiments on the real images demonstrate the effectiveness of our presented framework.

Improved Particle Swarm Optimization Approach for Vibration Vision Measurement

Traditional template matching-based motion estimation is a popular but time-consuming method for vibration vision measurement. In this study, the particle swarm optimization (PSO) algorithm is improved to solve this time-consumption problem. The convergence speed of the algorithm is increased using the adjacent frames search method in the particle swarm initialization process. A flag array is created to avoid repeated calculation in the termination strategy. The subpixel positioning accuracy is ensured by applying the surface fitting method. The robustness of the algorithm is ensured by applying the zero-mean normalized cross correlation. Simulation results demonstrate that the average extraction error of the improved PSO algorithm is less than 1%. Compared with the commonly used three-step search algorithm, diamond search algorithm, and local search algorithm, the improved PSO algorithm consumes the least number of search points. Moreover, tests on real-world image sequences show good estimation accuracy at very low computational cost. The improved PSO algorithm proposed in this study is fast, accurate, and robust, and is suitable for plane motion estimation in vision measurement.

Track color space-time interest points in video

Color Space-Time Interest Points (CSTIP) are among all the interesting low-level features which can be extracted from videos; they provide an efficient characterization of moving objects. The CSTIP are simple and can be used for video stabilization, camera motion estimation, and object tracking. In this paper, we show how the resulting features often reflect interesting events that can be used for a compact representation of video data as well as for tracking. To increase the robustness of CSTIP features extraction, we suggest a pre-processing step which is based on a Color Video Decomposition and can decompose the input images into a dynamic color texture and structure components. We compute the new Color Space Time Interest Points (CSTIP) associated to the dynamic color texture components by using the proposed algorithm of the detection of Color Space- Time Interest Points. The point tracker object tracks a set of Color Space-Time Interest Points using the robust Zero-Mean Normalized Cross-Correlation (ZNCC), feature-tracking algorithm. Experimental results are obtained from very different types of videos, namely sport videos and animation movies.

Application of 2D temperature measurement to a coal-fired furnace using CT-TDLAS

The measurement of temperature and species concentration in combustion fields is very significant for the development of highly efficient combustion technologies for energy conservation and emission reduction. There are various measurement technologies, including contact and non-contact measurement. Tunable diode laser absorption spectroscopy (TDLAS) technology is a proven non-contact method for detection of the temperature and species concentration by absorption measurement. To enable two-dimensional (2D) representation of temperature and species concentration in combustion fields, the TDLAS technology is usually combined with computed tomography (CT). The latter is, however, considerably new in combustion research, especially in the solid fuel reaction environment. In this paper, a 32-path 2D CT-TDLAS system for temperature measurement in a pilot-scale coal-fired furnace was developed. The accuracy of the CT algorithm in the reconstruction of 2D temperature distributions in different laser-path arrangements was first analyzed using the sum of squared difference (SSD) and zero-mean normalized cross-correlation (ZNCC) by comparing it to the 2D temperature distribution of a full-scale coal-fired furnace simulated using computational fluid dynamics (CFD). The accuracy was improved by 32-path reconstruction. The study was then progressed to investigate its measurement accuracy in a simple CH4–air burner configuration with rounded and rectangular cells as well as sensitivity for flame shift detection, whereby the reconstructed temperature distribution was compared to the temperature measured using a thermocouple. It is verified that this CT reconstruction was feasible for various measurement areas, even if the center of the flame was shifted. Finally, a 32-path 2D CT-TDLAS system with a rectangular-structured cell was developed and applied for temperature measurement in a Tenaga Nasional Berhad (TNB) Research’s pilot-scale coal-fired furnace. The 2D temperature distribution in the coal-fired furnace was reconstructed according to the experimental results. The potential of the CT-TDLAS for online 2D temperature measurement for actual applications is demonstrated.

Self-supervised monocular image depth learning and confidence estimation

We present a novel self-supervised framework for monocular image depth learning and confidence estimation. Our framework reduces the amount of ground truth annotation data required for training Convolutional Neural Networks (CNNs), which is often a challenging problem for the fast deployment of CNNs in many computer vision tasks. Our DepthNet adopts a novel fully differential patch-based cost function through the Zero-Mean Normalized Cross Correlation (ZNCC) to take multi-scale patches as matching and learning strategies. This approach greatly increases the accuracy and robustness of the depth learning. Whilst the proposed patch-based cost function naturally provides a 0-to-1 confidence, it is then used to self-supervise the training of a parallel network for confidence map learning and estimation by exploiting the fact that ZNCC is a normalized measure of similarity which can be approximated as the confidence of the depth estimation. Therefore, the proposed corresponding confidence map learning and estimation operate in a self-supervised manner and is a parallel network to the DepthNet. Evaluation on the KITTI depth prediction evaluation dataset and Make3D dataset show that our method outperforms the state-of-the-art results.

Well-Distributed Feature Extraction for Image Registration Using Histogram Matching

Image registration is a spatial alignment of corresponding images of the same scene acquired from different views, sensors, and time intervals. Especially, satellite image registration is a challenging task due to the high resolution of images. In addition, demands for high resolution satellite imagery are increased for more detailed and precise information in land planning, urban planning, and Earth observation. Commonly, feature-based methods are applied for image registration. In these methods, first control or key points are detected using feature detector such as scale-invariant feature transform (SIFT). The numbers and the distribution of these control points are important for the remaining steps of registration. These methods provide reasonable performance; however, they suffer from high computational cost and irregular distribution of control points. To overcome these limitations, we propose an area-based registration method using histogram matching and zero mean normalized cross-correlation (ZNCC). In multi-spectral satellite images, first, different spectral responses are adjusted by using histogram matching. Then, ZNCC is utilized to extract well-distributed control points. In addition, fast Fourier transform (FFT) and block-wise processing are applied to reduce the computational cost. The proposed method is evaluated through various input datasets. The results demonstrate its efficacy and accuracy in image registration.

Evaluation of Matching Costs for High-Quality Sea-Ice Surface Reconstruction from Aerial Images

Satellite remote sensing can be used effectively with a wide coverage and repeatability in large-scale Arctic sea-ice analysis. To produce reliable sea-ice information, satellite remote-sensing methods should be established and validated using accurate field data, but obtaining field data on Arctic sea-ice is very difficult due to limited accessibility. In this situation, digital surface models derived from aerial images can be a good alternative to topographical field data. However, to achieve this, we should discuss an additional issue, i.e., that low-textured surfaces on sea-ice can reduce the matching accuracy of aerial images. The matching performance is dependent on the matching cost and search window size used. Therefore, in order to generate high-quality sea-ice surface models, we first need to examine the influence of matching costs and search window sizes on the matching performance on low-textured sea-ice surfaces. For this reason, in this study, we evaluate the performance of matching costs in relation to changes of the search window size, using acquired aerial images of Arctic sea-ice. The evaluation concerns three factors. The first is the robustness of matching to low-textured surfaces. Matching costs for generating sea-ice surface models should have a high discriminatory power on low-textured surfaces, even with small search windows. To evaluate this, we analyze the accuracy, uncertainty, and optimal window size in terms of template matching. The second is the robustness of positioning to low-textured surfaces. One of the purposes of image matching is to determine the positions of object points that constitute digital surface models. From this point of view, we analyze the accuracy and uncertainty in terms of positioning object points. The last is the processing speed. Since the computation complexity is also an important performance indicator, we analyze the elapsed time for each of the processing steps. The evaluation results showed that the image domain costs were more effective for low-textured surfaces than the frequency domain costs. In terms of matching robustness, the image domain costs showed a better performance, even with smaller search windows. In terms of positioning robustness, the image domain costs also performed better because of the lower uncertainty. Lastly, in terms of processing speed, the PC (phase correlation) of the frequency domain showed the best performance, but the image domain costs, except MI (mutual information), were not far behind. From the evaluation results, we concluded that, among the compared matching costs, ZNCC (zero-mean normalized cross-correlation) is the most effective for sea-ice surface model generation. In addition, we found that it is necessary to adjust search window sizes properly, according to the number of textures required for reliable image matching on sea-ice surfaces, and that various uncertainties due to low-textured surfaces should be considered to determine the positions of object points.

Stereo Matching Methods for Imperfectly Rectified Stereo Images

Stereo matching has been under development for decades and is an important process for many applications. Difficulties in stereo matching include textureless regions, occlusion, illumination variation, the fattening effect, and discontinuity. These challenges are effectively solved in recently developed stereo matching algorithms. A new imperfect rectification problem has recently been encountered in stereo matching, and the problem results from the high resolution of stereo images. State-of-the-art stereo matching algorithms fail to exactly reconstruct the depth information using stereo images with imperfect rectification, as the imperfectly rectified image problems are not explicitly taken into account. In this paper, we solve the imperfect rectification problems, and propose matching stereo matching methods that based on absolute differences, square differences, normalized cross correlation, zero-mean normalized cross correlation, and rank and census transforms. Finally, we conduct experiments to evaluate these stereo matching methods using the Middlebury datasets. The experimental results show the proposed stereo matching methods can reduce error rate significantly for stereo images with imperfect rectification.

Reliability-guided Rayleigh backscattering spectrum correlation method for distributed strain measurements in optical fibres

We present a method of reliability-guided Rayleigh backscattering correlation for distributed strain measurements in optical fibres. In this method, a reference Rayleigh-backscattering-spectrum (RBS) range that is larger than the measurement RBS range is defined to extend the matching range. To obtain the best match between reference and measurement spectra, the zero-mean normalized cross correlation (ZNCC) is employed to evaluate the degree of similarity. The path for searching the maximum similarity matching pattern is guided using Newton’s iteration method. The reliability of the computed RBS shift is identified by the ZNCC coefficient distribution. The experiments show that the proposed method has high reliability in computing the RBS shift. Even at a relatively large strain (e.g. 5000 µϵ), the proposed method can stably demodulate the strain within a relative error of −1% and a spatial resolution of 1.6 cm over a 22-meter-long single-mode fibre. This shows that the proposed method has an advantage in regard to relatively large strain measurements.

Eye-to-Hand Robotic Visual Tracking Based on Template Matching on FPGAs

This paper presents a delta robotic visual-servoing tracking method using zero-mean normalized cross-correlation (ZNCC)-based grayscale template matching hardware core on field-programmable gate arrays (FPGAs). The concurrent FPGA-based ZNCC hardware core with cascading multiplication-accumulate (MAC) circuits is designed, which can largely reduce FPGA hardware resource consumptions. A compact optical imaging system with a front 45°-slant mirror and an optical filter film are proposed, which can efficiently filter out the background cluttered artifacts. The trajectory visual-servoing tracking and dynamic tracking experiments based on our built-up delta robotic visual tracking platform are implemented. The experimental results indicate that the presented FPGA-based embedded robotic visual tracking method can efficiently improve an object trajectory tracking performance.

Mosaic secret-fragment-visible data hiding for secure image transmission based on two-step energy matching

Recently, a new information hiding technology was proposed to hide a secret image into secret-fragment-visible mosaic image for secure image transmission. However, the previous method pays less attention to the block matching in the transform, and may obtain a low quality mosaic transform in some cases. In this paper, we propose to improve the secret-fragment-visible data hiding based the two-step energy function, which returns a weighted value of standard deviation and total variation of block. We match all the blocks according to the energy function and take the zero-mean normalized cross correlation (ZNCC) as the similarity measurement to optimize the matching. The interpolation optimization algorithm is used to stimulate the total ZNCC changes along with the parameter selection, and helps to solve the optimization quickly. Since more block's statistical properties are employed, the image blocks can be matched accurately, and a low distortion is introduced in the transform. Experimental results show that the proposed method can obtain a better visual quality of secret image after extraction and also requires less bits to encode the residuals.