Image Similarity Measure Research Articles

A Novel Method for Remaining Useful Life Prediction of Bearing Based on Spectrum Image Similarity Measures

Accurately predicting the remaining useful life (RUL) of bearing by analyzing vibration signals is challenging and meaningful. To address this issue, a novel method based on spectrum image similarity is proposed in this paper. First, spectrum images for the whole lifecycle data of reference bearings are obtained by performing fast Fourier transformation (FFT). Second, the similarity is calculated between the current monitored data of operating bearing and run-to-failure images of reference bearings. Then, the weights of reference bearings are derived based on the similarity measures. Finally, the RUL of the operating bearing is estimated with the weighted average of the RULs of referenced bearings. The proposed method is demonstrated based on 2012 PHM Data Challenge Competition data, which shows its effectiveness and practicality.

Subspace cross representation measure for robust face recognition with few samples

Similarity measure generally exerts a crucial role in face recognition. Recently, regression analysis based similarity measure mechanism has demonstrated significant potential in robust face recognition. Nevertheless, most existing regression methods are far from perfect under few samples due to the poor performance of spanning the individual subspace. Previous works have been noticed that the singular value decomposition (SVD) of facial image can generate a set of complete base of individual subspace. Then we present a novel and efficient image similarity measure model named subspace cross representation (SCR) measure for face recognition with few samples. The power of our proposed SCR stems from the following facts. One is that the complete base can weaken the dependence of linear regression method on the number of labeled samples. The other is the cross linear representation can effectively use two-dimensional geometric features generated by SVD to distinguish facial images. The validity of SCR is tested by a large amount of experiments on AR, CUHK Sketch, Extended Yale B databases, etc. The experimental results demonstrate that SCR achieves satisfactory recognition accuracy compared with other methods, under few sample condition.

A Weakly Supervised Framework for 2D/3D Vascular Registration Oriented to Incomplete 2D Blood Vessels

Registering preoperative 3D computed tomography (CT) to intraoperative 2D digital subtraction angiography (DSA) can complete the missing part of 2D vessels by 3D-2D projection, which is useful for the guidance of vascular interventional surgery. However, vessel excalation in DSAs remains hard for registration in clinical circumstances. To overcome this challenge, we proposed a weakly supervised 2D/3D vascular registration framework to improve the performance on DSAs with incomplete vessels. We combined a CNN regressor with a novel 3D-2D mask projection module to build the full network. Then a two-step training procedure is employed, including supervised pre-training on simulated images and weakly supervised finetuning on DSAs. In addition, a patch-based content loss is introduced in the finetuning step to give an appropriate similarity measure for images with incomplete vessels. We evaluated our method on both simulated and real images. Experiments prove that our proposed method is effective to improve the backbone regressor’s registration performance on incomplete vessels and has the ability to complete vessels in DSAs with high recall.

Colorizing Grayscale CT images of human lungs using deep learning methods

Image colorization refers to computer-aided rendering technology which transfers colors from a reference color image to grayscale images or video frames. Deep learning elevated notably in the field of image colorization in the past years. In this paper, we formulate image colorization methods relying on exemplar colorization and automatic colorization, respectively. For hybrid colorization, we select appropriate reference images to colorize the grayscale CT images. The colours of meat resemble those of human lungs, so the images of fresh pork, lamb, beef, and even rotten meat are collected as our dataset for model training. Three sets of training data consisting of meat images are analysed to extract the pixelar features for colorizing lung CT images by using an automatic approach. Pertaining to the results, we consider numerous methods (i.e., loss functions, visual analysis, PSNR, and SSIM) to evaluate the proposed deep learning models. Moreover, compared with other methods of colorizing lung CT images, the results of rendering the images by using deep learning methods are significantly genuine and promising. The metrics for measuring image similarity such as SSIM and PSNR have satisfactory performance, up to 0.55 and 28.0, respectively. Additionally, the methods may provide novel ideas for rendering grayscale X-ray images in airports, ferries, and railway stations.

Milling chatter monitoring based on sparse representation and image similarity measurement

It is well known that chatter is one of the main bottlenecks affecting the surface quality of workpieces and production efficiency during machining. In this paper, a new chatter monitoring methodology is proposed on the basis of sparse representation and image similarity measurement to recognise chatter in the manufacturing process. Due to its non-stationary nature with regard to chatter signals, variational mode decomposition (VMD) is utilised and timefrequency entropy (TFE) based on VMD is introduced to measure the complexity. Then, the overcomplete dictionary is pre-trained using the characteristic matrix image extracted from the multi-domain features, thus facilitating the description of chatter from various perspectives. Subsequently, the visualised sparse coefficient matrix is acquired from the trained dictionary and regarded as the reference image, in which detailed information can be obtained from the visualisation of the image. Next, an image similarity measurement method is applied to assess the similarity between the tested sparse coefficient image and the reference image, thereby considering the local and global quality maps such that a comprehensive index for chatter detection can be obtained to fuse the various features. Finally, to validate the proposed methodology, experimental chatter tests are conducted under different machining conditions. The results demonstrate that the chatter can be discriminated at the early stage of chatter development, thus leaving more time to take suppression measures.

MCP-Net: Inter-frame Motion Correction with Patlak Regularization for Whole-body Dynamic PET.

Inter-frame patient motion introduces spatial misalignment and degrades parametric imaging in whole-body dynamic positron emission tomography (PET). Most current deep learning inter-frame motion correction works consider only the image registration problem, ignoring tracer kinetics. We propose an inter-frame Motion Correction framework with Patlak regularization (MCP-Net) to directly optimize the Patlak fitting error and further improve model performance. The MCP-Net contains three modules: a motion estimation module consisting of a multiple-frame 3-D U-Net with a convolutional long short-term memory layer combined at the bottleneck; an image warping module that performs spatial transformation; and an analytical Patlak module that estimates Patlak fitting with the motion-corrected frames and the individual input function. A Patlak loss penalization term using mean squared percentage fitting error is introduced to the loss function in addition to image similarity measurement and displacement gradient loss. Following motion correction, the parametric images were generated by standard Patlak analysis. Compared with both traditional and deep learning benchmarks, our network further corrected the residual spatial mismatch in the dynamic frames, improved the spatial alignment of Patlak Ki/Vb images, and reduced normalized fitting error. With the utilization of tracer dynamics and enhanced network performance, MCP-Net has the potential for further improving the quantitative accuracy of dynamic PET. Our code is released at https://github.com/gxq1998/MCP-Net.

PTF‐SimCM: A Simple Contrastive Model with Polysemous Text Fusion for Visual Similarity Metric

Image similarity metric, also known as metric learning (ML) in computer vision, is a significant step in various advanced image tasks. Nevertheless, existing well‐performing approaches for image similarity measurement only focus on the image itself without utilizing the information of other modalities, while pictures always appear with the described text. Furthermore, those methods need human supervision, yet most images are unlabeled in the real world. Considering the above problems comprehensively, we present a novel visual similarity metric model named PTF‐SimCM. It adopts a self‐supervised contrastive structure like SimSiam and incorporates a multimodal fusion module to utilize textual modality correlated to the image. We apply a cross‐modal model for text modality rather than a standard unimodal text encoder to improve late fusion productivity. In addition, the proposed model employs Sentence PIE‐Net to solve the issue caused by polysemous sentences. For simplicity and efficiency, our model learns a specific embedding space where distances directly correspond to the similarity. Experimental results on MSCOCO, Flickr 30k, and Pascal Sentence datasets show that our model overall outperforms all the compared methods in this work, which illustrates that the model can effectively address the issues faced and enhance the performances on unsupervised visual similarity measuring relatively.

Unaligned Hyperspectral Image Fusion via Registration and Interpolation Modeling

In satellite remote sensing, the hyperspectral sensor acquires high-spectral-resolution and low-spatial-resolution hyperspectral images (HSIs). Conversely, the multispectral sensor acquires low-spectral-resolution and high-spatial-resolution multispectral images (MSIs). Thus, HSI and MSI fusion is required to promote both spatial and spectral resolutions. Currently, most algorithms are based on the assumption that the HSI and MSI are perfectly aligned. However, this is hardly achievable in real scenarios when the two sensors acquire images from different viewpoints. In this article, we propose a fusion algorithm that consists of two stages, i.e., image registration and image fusion. For image registration, we introduce the normalized edge difference (NED) for image similarity measure considering the different resolutions of the original images. For image fusion, we incorporate the interpolation process in the spatial degradation model to compensate for the interpolation error. Experimental results show that our algorithm performs better than the state of the arts for unaligned image fusion.

A Similarity Measure of Histopathology Images by Deep Embeddings.

Histopathology digital scans are large-size images that contain valuable information at the pixel level. Contentbased comparison of these images is a challenging task. This study proposes a content-based similarity measure for highresolution gigapixel histopathology images. The proposed similarity measure is an expansion of cosine vector similarity to a matrix. Each image is divided into same-size patches with a meaningful amount of information (i.e., contained enough tissue). The similarity is measured by the extraction of patchlevel deep embeddings of the last pooling layer of a pre-trained deep model at four different magnification levels, namely, 1x, 2.5x, 5x, and 10x magnifications. In addition, for faster measurement, embedding reduction is investigated. Finally, to assess the proposed method, an image search method is implemented. Results show that the similarity measure represents the slide labels with a maximum accuracy of 93.18% for top-5 search at 5x magnification.

Unsupervised Learning-Based CBCT-CT Deformable Image Registration for CBCT-Guided Abdominal Radiotherapy

Daily Cone beam CT (CBCT) imaging provides necessary anatomical information for accurate patient setup. Image quality of CBCT is usually far inferior to simulation CT scans. A workaround is to register the CT to the CBCT such that the contours and Hounsfield Unit (HU) values of the CT can be propagated to the CBCT. However, the inconsistent HU values across CT and CBCT make it less effective to use conventional image similarity measures. We aim to develop an unsupervised registration network to overcome this challenge in multimodal CT-CBCT image registration.We propose to integrate directional local structural similarity into an unsupervised learning framework to perform abdominal CT-CBCT image registration. Directional local structural similarity measures the image's self-similarity which reflects the underlying structural similarity regardless of the modality in use. The CBCT and CT images were separately processed to extract directional local structural similarity feature maps in different directions. We concatenated the directional local structural similarity feature maps and the original images as network input. Taking both the original images and their respective structural similarity feature maps as input allows the network to fully explore the potential correlations between CBCT and CT for accurate deformation vector field (DVF) prediction. Salient features learnt through previous iterations were highlighted by attention gates across layers to expedite the learning process. A 3D bicubic interpolation was used to up-sample and smooth the predicted DVF. We performed a leave-one-out cross validation with an image dataset of 45 patients to evaluate the proposed registration method. Normalized cross correlation (NCC) and target registration error (TRE) between CBCT and deformed CT were calculated to quantify the registration accuracy.Our results show that the alignment between the abdominal soft tissues has been greatly improved after registration for all patients. The mean and standard deviation of NCC and TRE were 0.97 (range 0.95-0.99) and 1.88 (range 1.03-2.67) mm. The proposed network allows for many datasets to be used as training datasets since ground truth DVF is not needed for the training process. The proposed network can predict a final DVF via a single forward prediction, which is faster than the conventional iterative registration algorithms.We have developed a novel unsupervised multimodal image registration method for CT-CBCT abdominal image registration, which does not need ground truth DVF for training, and demonstrated its feasibility. Taking both CBCT and CT images and their respective directional local structural similarity features as input, the proposed network performs direct DVF prediction to register the abdominal CT to CBCT images. This tool will be useful for future CBCT-guided radiotherapy of abdominal malignancies.

IMGTR: Image-triangle based multi-view 3D reconstruction for urban scenes

Multi-view depth map reconstruction is a popular approach to generate 3D information with good flexibility and scalability. However, texture-weak regions and repeated textures challenge these methods in urban scenes. To address this need, this paper proposes an image-triangle based multi-view 3D reconstruction (IMGTR) method. Starting from constructing a density-adaptive image triangulation for each image, the main procedure is to determine the corresponding object plane for each image triangle under an objective function consisting of the image similarity measure, the smoothness constraint and the continuity constraint on the edges between adjacent triangles. Qualitative and quantitative experiments show that the proposed method can reconstruct complex urban structures more accurate and achieve higher fidelity in planar urban structures than some recently released multi-view stereo algorithms. Specifically, for the Vaihingen dataset IMGTR achieves an average improvement of 4.27% compared to PMVD, 50.94% to SURE and 54.76% to COLMAP in position accuracy.

Thai Sign Language Image Recognition for the Hearing-Impaired using Radial Inverse Force Histogram combined with Max-Min Boundary

Sign language image recognition is also a very interesting research topic. Because it can be applied to help normal people understand and use it as a communication tool for the hearing impaired. The objectives of this research were to: 1) study and analyze Thai Sign Language image recognition image data for hearing impaired, 2) develop Thai Sign Language image recognition system for hearing impaired by using new techniques, and 3) measure the efficiency of Thai sign language image recognition for hearing impaired using Radial Inverse Force Histogram and the Maximum and Minimum boundary values. The results of the research were as follows: 1) The study and analysis of image data of Thai Sign Language Image Recognition In this research, 62,694 sign language images were used, divided into 2 parts: 1) American Sign Language images, which consisted of 36 groups of images, namely 26 groups of letters (AZ) and 10 groups of numbers (0-9), and Part 2) Picture of Thai Sign Language consisting of 61 groups of images, including 44 groups of letters (ก-ฮ), 7 groups of vowels and 10 groups of numbers (0-9). Each group of pictures is rotated, enlarged, and Image promotion There were 6 sub-groups of images in various forms, divided into 2 parts: 70% of the images for training and 30% of the images for testing. 3) The results of measuring the efficiency of image recognition. It is divided into two parts: American Sign Language Image Recognition and Thai Sign Language Image Recognition. Compared with the Angular histogram method, the mean image accuracy was 0.86, the recall of the mean American sign language was 0.91, and the accuracy of the Thai Sign Language was 0.78. The recognition performance for Thai Sign Language images averaged 0.89, while the recognition efficiency was achieved when using radial inverse force histograms in combination with image similarity measurements with maximum-minimum boundary values. Accuracy for Mean American Sign Language was 0.99 and Remembrance for Mean American Sign was 1.00, while Accuracy for Mean Thai Sign Language was 0.89. Mean Remembrance for Thai Sign Language was 0.96. The results of the visual recognition performance measurement of both the American Sign Language and the Thai Sign Language images were very good compared to the Angular Histogram method.

Matching on Sets: Conquer Occluded Person Re-identification Without Alignment

Occluded person re-identification (re-ID) is a challenging task as different human parts may become invisible in cluttered scenes, making it hard to match person images of different identities. Most existing methods address this challenge by aligning spatial features of body parts according to semantic information (e.g. human poses) or feature similarities but this approach is complicated and sensitive to noises. This paper presents Matching on Sets (MoS), a novel method that positions occluded person re-ID as a set matching task without requiring spatial alignment. MoS encodes a person image by a pattern set as represented by a `global vector’ with each element capturing one specific visual pattern, and it introduces Jaccard distance as a metric to compute the distance between pattern sets and measure image similarity. To enable Jaccard distance over continuous real numbers, we employ minimization and maximization to approximate the operations of intersection and union, respectively. In addition, we design a Jaccard triplet loss that enhances the pattern discrimination and allows to embed set matching into deep neural networks for end-to-end training. In the inference stage, we introduce a conflict penalty mechanism that detects mutually exclusive patterns in the pattern union of image pairs and decreases their similarities accordingly. Extensive experiments over three widely used datasets (Market1501, DukeMTMC and Occluded-DukeMTMC) show that MoS achieves superior re-ID performance. Additionally, it is tolerant of occlusions and outperforms the state-of-the-art by large margins for Occluded-DukeMTMC.

Fuzzy Based Fast Non Local Mean Filter to Denoise Rician Noise

Abstract MRI images suffer from a type of multiplicative noise called Rician noise. The traditional smoothing filters are not effective to reduce this type of noise. Non local Means (NLM) filters with bias correction is a promising technique for such type of signal dependent noise. The NLM denoising is based on self-similarity and is computationally very expensive. Many researchers are working to improve its time complexity. In most cases there is a tradeoff between speed and quality of denoised image. This paper proposes a speed enhanced NLM method that use integral image and fuzzy Jacord similarity measure to find the self-similarity of sub windows. This helps to find the weights of similar pixel at a faster rate than the traditional NLM algorithm and more accurately than the existing fast NLM method. Consequently, these similar pixels are used to generate the noise free pixels. At the end the conventional bias subtraction method is used as post processing step. The proposed scheme is tested with the standard brain web data set with different noise levels and compared with existing Fast NLM techniques and basic NLM using Root mean square error (RMSE), peak signal noise ratio (PSNR), Structure similar index (SSIM), quality index and computational time parameters methods. The proposed method gives better result than existing Fast NLM technique with high density Rician noise in the image and is 20 times faster than traditional NLM.

Robust Visual Place Recognition in Changing Environments Using Improved DTW

Recently, the methods based on Convolutional Neural Networks (CNNs) have achieved state-of-the-art performance in visual place recognition. CNN is a class of multilayer perceptrons, but unlike common multilayer perceptrons that it is not usually fully connected networks. It can acquire more general image features and make the image processing computationally manageable through filtering the connections by proximity. In this paper, we utilize the deep features generated by CNNs and the dynamic time warping (DTW) algorithm for image sequence place recognition. We propose a novel image similarity measurement, which is derived from cosine distance and can better distinguish match and mismatch. Meanwhile, we improve the DTW algorithm to design a local matching method that can reduce time complexity from O(n3) to O(n). To test the proposed method, four datasets (Nordland, Gardens Point, St. Lucia, and UoA datasets) are used as benchmarks; using two traverses in each dataset with one for reference and the other for testing. The results show high precision-recall characteristics of our method in the cases of severe appearance changes. Besides, our method achieves substantial improvements over the methods using the deep feature representations of a single image for recognition, which reflects that the spatiotemporal information contained in the image sequence is significant for the task of visual place recognition. Moreover, the proposed method also shows to outperform the classical sequence-based method SeqSLAM.

Deep learning‐based X‐ray inpainting for improving spinal 2D‐3D registration

Two-dimensional (2D)-3D registration is challenging in the presence of implant projections on intraoperative images, which can limit the registration capture range. Here, we investigate the use of deep-learning-based inpainting for removing implant projections from the X-rays to improve the registration performance. We trained deep-learning-based inpainting models that can fill in the implant projections on X-rays. Clinical datasets were collected to evaluate the inpainting based on six image similarity measures. The effect of X-ray inpainting on capture range of 2D-3D registration was also evaluated. The X-ray inpainting significantly improved the similarity between the inpainted images and the ground truth. When applying inpainting before the 2D-3D registration process, we demonstrated significant recovery of the capture range by up to 85%. Applying deep-learning-based inpainting on X-ray images masked by implants can markedly improve the capture range of the associated 2D-3D registration task.

Fuzzy based fast non local mean filter to denoise Rician noise

MRI images suffer from a type of multiplicative noise called Rician noise. The traditional smoothing filters are not effective to reduce this type of noise. Non local Means (NLM) filters with bias correction is a promising technique for such type of signal dependent noise. The NLM denoising is based on self-similarity and is computationally very expensive. Many researchers are working to improve its time complexity. In most cases there is a tradeoff between speed and quality of denoised image. This paper proposes a speed enhanced NLM method that use integral image and fuzzy Jacord similarity measure to find the self-similarity of sub windows. This helps to find the weights of similar pixel at a faster rate than the traditional NLM algorithm and more accurately than the existing fast NLM method. Consequently, these similar pixels are used to generate the noise free pixels. At the end the conventional bias subtraction method is used as post processing step. The proposed scheme is tested with the standard brain web data set with different noise levels and compared with existing Fast NLM techniques and basic NLM using Root mean square error (RMSE), peak signal noise ratio (PSNR), Structure similar index (SSIM), quality index and computational time parameters methods. The proposed method gives better result than existing Fast NLM technique with high density Rician noise in the image and is 20 times faster than traditional NLM.

Weakly supervised video object segmentation initialized with referring expression

With the aid of one manually annotated frame, One-Shot Video Object Segmentation (OSVOS) uses a CNN architecture to tackle the problem of semi-supervised video object segmentation (VOS). However, annotating a pixel-level segmentation mask is expensive and time-consuming. To alleviate the problem, we explore a language interactive way of initializing semi-supervised VOS and run the semi-supervised methods into a weakly supervised mode. Our contributions are two folds: (i) we propose a variant of OSVOS initialized with referring expressions (REVOS), which locates a target object by maximizing the matching score between all the candidates and the referring expression; (ii) segmentation performance of semi-supervised VOS methods varies dramatically when selecting different frames for annotation. We present a strategy of the best annotation frame selection by using image similarity measurement. Meanwhile, we first to propose a multiple frame annotation selection strategy for initialization of semi-supervised VOS with more than one annotated frames. Finally we evaluate our method on DAVIS-2016 dataset, and experimental results show that REVOS achieves similar performance (79.94% measured by average IoU) compared with OSVOS (80.1%). Although current REVOS implementation is specific to the method of one-shot video object segmentation, it can be more widely applicable to other semi-supervised VOS methods.

Subspace transform induced robust similarity measure for facial images

Similarity measure has long played a critical role and attracted great interest in various areas such as pattern recognition and machine perception. Nevertheless, there remains the issue of developing an efficient two-dimensional (2D) robust similarity measure method for images. Inspired by the properties of subspace, we develop an effective 2D image similarity measure technique, named transformation similarity measure (TSM), for robust face recognition. Specifically, the TSM method robustly determines the similarity between two well-aligned frontal facial images while weakening interference in the face recognition by linear transformation and singular value decomposition. We present the mathematical features and some odds to reveal the feasible and robust measure mechanism of TSM. The performance of the TSM method, combined with the nearest neighbor rule, is evaluated in face recognition under different challenges. Experimental results clearly show the advantages of the TSM method in terms of accuracy and robustness.

Label-driven magnetic resonance imaging (MRI)-transrectal ultrasound (TRUS) registration using weakly supervised learning for MRI-guided prostate radiotherapy

Registration and fusion of magnetic resonance imaging (MRI) and transrectal ultrasound (TRUS) of the prostate can provide guidance for prostate brachytherapy. However, accurate registration remains a challenging task due to the lack of ground truth regarding voxel-level spatial correspondence, limited field of view, low contrast-to-noise ratio, and signal-to-noise ratio in TRUS. In this study, we proposed a fully automated deep learning approach based on a weakly supervised method to address these issues. We employed deep learning techniques to combine image segmentation and registration, including affine and nonrigid registration, to perform an automated deformable MRI-TRUS registration. To start with, we trained two separate fully convolutional neural networks (CNNs) to perform a pixel-wise prediction for MRI and TRUS prostate segmentation. Then, to provide the initialization of the registration, a 2D CNN was used to register MRI-TRUS prostate images using an affine registration. After that, a 3D UNET-like network was applied for nonrigid registration. For both the affine and nonrigid registration, pairs of MRI-TRUS labels were concatenated and fed into the neural networks for training. Due to the unavailability of ground-truth voxel-level correspondences and the lack of accurate intensity-based image similarity measures, we propose to use prostate label-derived volume overlaps and surface agreements as an optimization objective function for weakly supervised network training. Specifically, we proposed a hybrid loss function that integrated a Dice loss, a surface-based loss, and a bending energy regularization loss for the nonrigid registration. The Dice and surface-based losses were used to encourage the alignment of the prostate label between the MRI and the TRUS. The bending energy regularization loss was used to achieve a smooth deformation field. Thirty-six sets of patient data were used to test our registration method. The image registration results showed that the deformed MR image aligned well with the TRUS image, as judged by corresponding cysts and calcifications in the prostate. The quantitative results showed that our method produced a mean target registration error (TRE) of 2.53 ± 1.39 mm and a mean Dice loss of 0.91 ± 0.02. The mean surface distance (MSD) and Hausdorff distance (HD) between the registered MR prostate shape and TRUS prostate shape were 0.88 and 4.41 mm, respectively. This work presents a deep learning-based, weakly supervised network for accurate MRI-TRUS image registration. Our proposed method has achieved promising registration performance in terms of Dice loss, TRE, MSD, and HD.