Visual Similarity Measure Research Articles

Infrared-Visible Synthetic Data from Game Engine for Image Fusion Improvement

Limited by the shooting scenes and angles of fixed cameras, the existing datasets generally lack many detailed pedestrian models in diverse scenarios. Existing deep learning-based image fusion methods, for this reason, bring about overfitting or insufficient information of fusion results in varying degrees. To address this challenge, A new infrared-visible pedestrian synthetic dataset (GIVF) with a synthetic data tagger (GSDT) is constructed and an improved end-to-end image fusion network (FSGAN) is proposed to validate infrared and visible fusion. In the model, the method uses an auxiliary network to extract features that complement the cascade network of the main path, effectively improving the ability to extract pedestrian texture details. Experimental results show that FSGAN can be well applied to GIVF. By conducting extensive comparative experiments with eight state-of-the-art image fusion methods. FSGAN shows better performance than those comparison methods, especially in the two evaluation indexes visual information fidelity (VIF) and structural similarity measurement (SSIM). Besides, by comparing the quantitative analysis results of various methods, and evaluating the fusion results of real images in complex environments on other three datasets, we conclude that FSGAN can be better applied to GIVF datasets than other popular methods, and has outstanding performance in generalization.

Advanced Image Processing Techniques for Medical Image Retrieval Using Visual Features and Distance Measures

Recently, there has been a rapid increase in the volume of medical image repositories because of the maximized utilization of digitized image data in hospitals. As a result, there is a complexity in querying and handling such large databases, which has led to the development of a novel Content-Based Medical Image Retrieval (CBMIR) technique. In this work, the CBMIR technique is developed using three types of visual features (i.e., color, texture and shape) along with 12 distance measures, and is optimized with the Crossover-Gravitational Search Algorithm (CRO-GSA). For ease, we named this technique Content-Based Medical Image Retrieval using Visual features with CRO-GSA (CBMIRVC). The initial step of our CBMIRVC technique is to extract three types of visual features from the images. Then, each type of feature is employed with a suitable distance measure, which is used for the computation of image similarities between the medical images in the database and a provided query image. The optimization of our proposed CBMIRVC technique is done by the CRO-GSA algorithm. This algorithm optimizes the CBMIRVC technique by identifying the optimal combinations of visual features and similarity measures. Additionally, optimal weights are computed in accordance with the three types of features for the three similarities. Experimental validations were done using retrieved cancer-related and endoscopic color images from databases holding numerous image categories. The experimental outcome shows that our proposed CBMIRVC technique outperforms other conventional image retrieval techniques by effectively retrieving similar images from large databases.

Fully Connected Generative Adversarial Network for Human Activity Recognition

Conditional Generative Adversarial Networks (CGAN) have shown great promise in generating synthetic data for sensor-based activity recognition. However, one key issue concerning existing CGAN is the design of the network architecture that affects sample quality. This study proposes an effective CGAN architecture that synthesizes higher quality samples than state-of-the-art CGAN architectures. This is achieved by combining convolutional layers with multiple fully connected networks in the generator’s input and discriminator’s output of the CGAN. We show the effectiveness of the proposed approach using elderly data for sensor-based activity recognition. Visual evaluation, similarity measure, and usability evaluation is used to assess the quality of generated samples by the proposed approach and validate its performance in activity recognition. In comparison to the state-of-the-art CGAN, the visual evaluation and similarity measure demonstrate that the proposed models’ synthetic data more accurately represents actual data and creates more variations in each synthetic data than the state-of-the-art approach respectively. The experimental stages of the usability evaluation, on the other hand, show a performance gain of 2.5%, 2.5%, 3.1%, and 4.4% over the state-of-the-art CGAN when using synthetic samples by the proposed architecture.

Data-enabled sketch search and retrieval for visual design stimuli generation

Abstract Access to vast datasets of visual and textual materials has become significantly easier. How to take advantage of the conveniently available data to support creative design activities remains a challenge. In the phase of idea generation, the visual analogy is considered an effective strategy to stimulate designers to create innovative ideas. Designers can read useful information off vague and incomplete conceptual visual representations, or stimuli, to reach potential visual analogies. In this paper, a computational framework is proposed to search and retrieve visual stimulation cues, which is expected to have the potential to help designers generate more creative ideas by avoiding visual fixation. The research problems include identifying and detecting visual similarities between visual representations from various categories and quantitatifying the visual similarity measures serving as a distance metric for visual stimuli search and retrieval. A deep neural network model is developed to learn a latent space that can discover visual relationships between multiple categories of sketches. In addition, a top cluster detection-based method is proposed to quantify visual similarity based on the overlapped magnitude in the latent space and then effectively rank categories. The QuickDraw sketch dataset is applied as a backend for evaluating the functionality of our proposed framework. Beyond visual stimuli retrieval, this research opens up new opportunities for utilizing extensively available visual data as creative materials to benefit design-by-analogy.

Perceptual similarity measurement based on generative adversarial neural networks in graphics design

Measuring the similarity between images is of paramount importance in computer vision. However, the commonly used pixelwise similarity metrics do not match well with perceptual similarity. The purpose of this paper is to propose a visual similarity measurement method, which can be effectively used for plagiarism detection in graphic design. Plagiarism detection of designs refers to the identification and determination of major similarities. It is difficult to carry out the similarity learning process in traditional deep neural network due to the insufficient of training samples. To overcome this problem, a novel scheme is proposed for measuring perceptual similarity of graphics by using a constraint Generative Adversarial Network (GAN) model. The generator of GAN is used to create similar graphics following the common plagiarism features of logo design. Unlike the traditional discriminator which judges the authenticity of the generated image and the original image, the modified discriminator is used to calculate the perceptual similarity of the graphics pair. In graphics design, plagiarism mainly focuses on the changes of shape, color and style, which has certain cognitive subjectivity. Therefore, design experts were invited to participate in a group of cognitive analysis experiments. A perceptual constraint model is established to limit the generation of plagiarized graphics according to “design and visual rationality”. Promising results demonstrate that the proposed method can be used for plagiarism detection of logo design. Given its effectiveness and conceptual simplicity, I hope it can serve as a baseline and contribute to the future research on plagiarism detection of artworks.

A Visual Comparison Method and Similarity Measure for Probabilistic Linguistic Term Sets and Their Applications in Multi-criteria Decision Making

The probabilistic linguistic term sets (PLTSs) are very powerful in solving the multi-criteria decision-making (MCDM) problems. The previous comparison methods associated with PLTSs are finite and unreasonable. Hence, developing a more effective way to compare PLTSs and proposing a reasonable decision-making method to cope with MCDM problems are important work of this paper. Firstly, a new possibility degree is proposed and a visual comparison method is given to present the process of comparing PLTSs. Subsequently, a similarity measure for PLTSs is also proposed to make up for the lack of similarity measure. Combining the new comparison method with the similarity measure, the TOPSIS is extended to solve real-life problems under probabilistic linguistic environment. Finally, a numerical example considers the selection of public opinion monitoring systems and the comparative analyses are shown to illustrate the effectiveness of the proposed method.

SVS-JOIN: Efficient Spatial Visual Similarity Join for Geo-Multimedia

In the big data era, massive amount of multimedia data with geo-tags has been generated and collected by smart devices equipped with mobile communications module and position sensor module. This trend has put forward higher request on large-scale geo-multimedia retrieval. Spatial similarity join is one of the significant problems in the area of spatial database. Previous works focused on spatial textual document search problem, rather than geo-multimedia retrieval. In this paper, we investigate a novel geo-multimedia retrieval paradigm named spatial visual similarity join (SVS-JOIN for short), which aims to search similar geo-image pairs in both aspects of geo-location and visual content. Firstly, the definition of SVS-JOIN is proposed and then we present the geographical similarity and visual similarity measurement. Inspired by the approach for textual similarity join, we develop an algorithm named SVS-JOIN $_{B}$ by combining the PPJOIN algorithm and visual similarity. Besides, an extension of it named SVS-JOIN $_{G}$ is developed, which utilizes spatial grid strategy to improve the search efficiency. To further speed up the search, a novel approach called SVS-JOIN $_{Q}$ is carefully designed, in which a quadtree and a global inverted index are employed. Comprehensive experiments are conducted on two geo-image datasets and the results demonstrate that our solution can address the SVS-JOIN problem effectively and efficiently.

Compressed sensing based feature fusion for image retrieval

In the standard vector of locally aggregated descriptors (VLAD) model, the residual norm of each descriptor with corresponding center to the VLAD vector varies significantly, thus the visual similarity measure based on norm measurement will be corrupted. To address the problem, a weighted-residual is used in this paper to balance the distribution of residual norms of all descriptors in the VLAD vector to a certain degree. Moreover, to improve retrieval accuracy, local gradient and local color features of images are extracted and used to our proposed weighted VLAD method. Also, the global structural features and multiple weighted VLAD vectors are fused to represent an image. In order to better fuse these different image features, due to the advantage of collecting key information of original signal by compressed sensing (CS), it is used as a converter to transform the various features into a same feature subspace. Our proposed method is evaluated on three benchmark datasets, i.e., Holidays, Ukbench and Oxford5k. Experimental results show that our proposed method achieves good performances compared with other methods.

Moving Object Detection via Color Optical Flow and Visual Objectness Measurement

How to find the moving object in the dynamic scene is one of the core problems in computer vision. In order to accurately detect moving objects in the video surveillance scene, optical flow algorithm and visual saliency detection technology had been researched. A moving object detection algorithm via color optical flow and visual similarity measurement is proposed in this paper. Firstly, the optical flow is detected on the visual significant color channel according to the opponent color theory, and the rough position of moving object is located. Then, two objectness measurements are defined to consider whether a region is an object in the precise positioning stage. Finally, the sliding window is used to mark the boundary box of the moving object in the sequence image. Experimental results show that the proposed algorithm is better than the classical moving object detection algorithm.

Video tamper detection based on multi-scale mutual information

Video frame manipulation has become commonplace with the growing easy access to powerful computing abilities. One of the most common types of video frame tampers is the copy-paste tamper, wherein a region from a video frame is replaced with another region from the same frame. In order to improve the robustness of passive video tampering detection, we propose a content-based video similarity tamper passive blind detection algorithm based on multi-scale normalized mutual information which can implement video frame copy, frame insertion and frame deletion tamper detection. The detail implementation of the proposed algorithm consists of multi-scale content analysis, single-scale content similarity measure, multi-scale content similarity measure, and tampering positioning. Firstly, we get the scales of the visual content of the video frame using Gaussian pyramid transform; Secondly, to measure the similarity of single-scale visual content, we define adjacent normalized mutual information of two frames according to information theory; Thirdly, we construct the multi-scale normalized mutual information descriptors to achieve the multi-scale visual content similarity measure of adjacent two frames using a linear combination. Finally, we use the local outlier isolated factor detection algorithm to detect the position of the video tampering. Experimental results show that the proposed approach can not only detect the video frame tampering position of delete, copy, and insert effectively, but also can detect the tampering of different and homology video encoding formats. We obtain a feature detecting accuracy in excess of 93% and detection rate of 96% across post processing operations, and are able to detect the delete, copy, and insert regions with a high true positive rate and lower false positive rate than the existing time field tamper detection methods.

Graph-based video sequence matching using dominant colour graph profile (DCGP)

This paper presents a fast and effective technique for videos’ visual similarity detection and measurement using compact fixed-length signatures. The proposed technique (dominant colour graph profile DCGP) extracts and encodes the spatio-temporal information of a given video shot into a graph-based structure (tree) that fully captures this vital information. The graph structured properties are utilized to construct a fixed-length video signature of 112 decimal values per video shot. The encoded spatio-temporal information is extracted following channelling each video frame into a block-based structure, where the positions of respective blocks are tracked across video frames and encoded into multiple DCGP trees. The proposed technique provides a high matching speed (>2000 fps) and robust retrieval performance. The experiments on various standard and challenging datasets shows the framework’s robust performance, in terms of both, retrieval and computational performances.

Video similarity detection using fixed-length Statistical Dominant Colour Profile (SDCP) signatures

This paper presents a fast and effective technique for videos’ visual similarity detection and measurement using compact fixed-length signatures. The proposed technique facilitates for building real-time and scalable video matching/retrieval systems through generating a representative signature for a given video shot. The generated signature (Statistical Dominant Colour Profile, SDCP) effectively encodes the colours’ spatio-temporal patterns in a given shot, towards a robust real-time matching. Furthermore, the SDCP signature is engineered to better address the visual similarity problem, through its relaxed representation of shot contents. The compact fixed-length aspect of the proposed signature is the key to its high matching speed (>1000 fps) compared to the current techniques that relies on exhaustive processing, such as dense trajectories. The SDCP signature encodes a given video shot with only 294 values, regardless of the shot length, which facilitates for speedy signature extraction and matching. To maximize the benefit of the proposed technique, compressed-domain videos are utilized as a case study following their wide availability. However, the proposed technique avoids full video decompression and operates on tiny frames, rather than full-size decompressed frames. This is achievable through using the tiny DC-images sequence of the MPEG compressed stream. The experiments on various standard and challenging datasets (e.g. UCF101 13k videos) shows the technique’s robust performance, in terms of both, retrieval ability and computational performances.

Precise shape matching of large shape datasets using hybrid approach

Precise and fast shape matching and retrieval from very large datasets is a challenging task because of the existence of many distortions such as noise, occlusion and affine distortions. In this paper, we aim to propose a time-saving and effective shape matching and retrieval framework, that employs pruning which will enable online shape retrieval from extremely large datasets. First, using a hierarchical tree-based structure supporting parallel processing and efficient feature descriptors, irrelevant shapes are pruned and a subset of shapes relevant to the query is selected, then using more sophisticated feature descriptors, accurate retrieval is performed. Contour representation of an object is considered as most significant visual shape similarity measure by the humans. Using boundary information, we generate two simplified and efficient feature descriptors for fast pruning, and a sophisticated feature descriptor for effective and accurate retrieval. Tests performed on standard datasets unveil that the proposed technique is computationally more efficient than the state-of-the-art techniques while maintaining comparable matching and retrieval performance. Its performance is scalable for huge datasets and is robust against affine transformations, articulations and occlusion.

Adaptive region matching for region‐based image retrieval by constructing region importance index

This study deals with the problem of similarity matching in region‐based image retrieval (RBIR). A novel visual similarity measurement called adaptive region matching (ARM) has been developed. For decreasing negative influence of interference regions and important information loss simultaneously, a region importance index is constructed and semantic meaningful region (SMR) is introduced. Moreover, ARM automatically performs SMR‐to‐image matching or image‐to‐image matching. Extensive experiments on Corel‐1000, Caltech‐256 and University of Washington (UW) databases demonstrate the authors proposed ARM is more flexible and more efficient than the existing visual similarity measurements that were originally developed for RBIR.

Learning a hybrid similarity measure for image retrieval

Learning similarity measure from relevance feedback has become a promising way to enhance the image retrieval performance. Existing approaches mainly focus on taking short-term learning experience to identify a visual similarity measure within a single query session, or applying long-term learning methodology to infer a semantic similarity measure crossing multiple query sessions. However, there is still a big room to elevate the retrieval effectiveness, because little is known in taking the relationship between visual similarity and semantic similarity into account. In this paper, we propose a novel hybrid similarity learning scheme to preserve both visual and semantic resemblance by integrating short-term with long-term learning processes. Concretely, the proposed scheme first learns a semantic similarity from the users' query log, and then, taking this as prior knowledge, learns a visual similarity from a mixture of labeled and unlabeled images. In particular, unlabeled images are exploited for the relevant and irrelevant classes differently and the visual similarity is learned incrementally. Finally, a hybrid similarity measure is produced by fusing the visual and semantic similarities in a nonlinear way for image ranking. An empirical study shows that using hybrid similarity measure for image retrieval is beneficial, and the proposed algorithm achieves better performance than some existing approaches.

Closing the loop in cortically-coupled computer vision: a brain–computer interface for searching image databases

We describe a closed-loop brain–computer interface that re-ranks an image database by iterating between user generated ‘interest’ scores and computer vision generated visual similarity measures. The interest scores are based on decoding the electroencephalographic (EEG) correlates of target detection, attentional shifts and self-monitoring processes, which result from the user paying attention to target images interspersed in rapid serial visual presentation (RSVP) sequences. The highest scored images are passed to a semi-supervised computer vision system that reorganizes the image database accordingly, using a graph-based representation that captures visual similarity between images. The system can either query the user for more information, by adaptively resampling the database to create additional RSVP sequences, or it can converge to a ‘done’ state. The done state includes a final ranking of the image database and also a ‘guess’ of the user's chosen category of interest. We find that the closed-loop system's re-rankings can substantially expedite database searches for target image categories chosen by the subjects. Furthermore, better reorganizations are achieved than by relying on EEG interest rankings alone, or if the system were simply run in an open loop format without adaptive resampling.

Similarly shaped letters evoke similar colors in grapheme–color synesthesia

Grapheme–color synesthesia is a neurological condition in which viewing numbers or letters (graphemes) results in the concurrent sensation of color. While the anatomical substrates underlying this experience are well understood, little research to date has investigated factors influencing the particular colors associated with particular graphemes or how synesthesia occurs developmentally. A recent suggestion of such an interaction has been proposed in the cascaded cross-tuning (CCT) model of synesthesia, which posits that in synesthetes connections between grapheme regions and color area V4 participate in a competitive activation process, with synesthetic colors arising during the component-stage of grapheme processing. This model more directly suggests that graphemes sharing similar component features (lines, curves, etc.) should accordingly activate more similar synesthetic colors. To test this proposal, we created and regressed synesthetic color-similarity matrices for each of 52 synesthetes against a letter-confusability matrix, an unbiased measure of visual similarity among graphemes. Results of synesthetes’ grapheme–color correspondences indeed revealed that more similarly shaped graphemes corresponded with more similar synesthetic colors, with stronger effects observed in individuals with more intense synesthetic experiences (projector synesthetes). These results support the CCT model of synesthesia, implicate early perceptual mechanisms as driving factors in the elicitation of synesthetic hues, and further highlight the relationship between conceptual and perceptual factors in this phenomenon.

A temporal precedence based clustering method for gene expression microarray data.

BackgroundTime-course microarray experiments can produce useful data which can help in understanding the underlying dynamics of the system. Clustering is an important stage in microarray data analysis where the data is grouped together according to certain characteristics. The majority of clustering techniques are based on distance or visual similarity measures which may not be suitable for clustering of temporal microarray data where the sequential nature of time is important. We present a Granger causality based technique to cluster temporal microarray gene expression data, which measures the interdependence between two time-series by statistically testing if one time-series can be used for forecasting the other time-series or not.ResultsA gene-association matrix is constructed by testing temporal relationships between pairs of genes using the Granger causality test. The association matrix is further analyzed using a graph-theoretic technique to detect highly connected components representing interesting biological modules. We test our approach on synthesized datasets and real biological datasets obtained for Arabidopsis thaliana. We show the effectiveness of our approach by analyzing the results using the existing biological literature. We also report interesting structural properties of the association network commonly desired in any biological system.ConclusionsOur experiments on synthesized and real microarray datasets show that our approach produces encouraging results. The method is simple in implementation and is statistically traceable at each step. The method can produce sets of functionally related genes which can be further used for reverse-engineering of gene circuits.

Spherical Correlation of Visual Representations for 3D Model Retrieval

In recent years we have seen a tremendous growth in the amount of freely available 3D content, in part due to breakthroughs for 3D model design and acquisition. For example, advances in range sensor technology and design software have dramatically reduced the manual labor required to construct 3D models. As collections of 3D content continue to grow rapidly, the ability to perform fast and accurate retrieval from a database of models has become a necessity. At the core of this retrieval task is the fundamental challenge of defining and evaluating similarity between 3D shapes. Some effective methods dealing with this challenge consider similarity measures based on the visual appearance of models. While collections of rendered images are discriminative for retrieval tasks, such representations come with a few inherent limitations such as restrictions in the image viewpoint sampling and high computational costs. In this paper we present a novel algorithm for model similarity that addresses these issues. Our proposed method exploits techniques from spherical signal processing to efficiently evaluate a visual similarity measure between models. Extensive evaluations on multiple datasets are provided.

A Real-Time Intrasubject Elastic Registration Algorithm for Dynamic 2-D Ultrasound Images

A new dynamic registration technique is designed to elastically align a sequence of 2-D ultrasound (US) images. The proposed algorithm tracks features over an image sequence in real-time, as opposed to our previous technique which registers images without utilizing prior information. The registration results were evaluated using a customized validation framework for elastic US registration algorithms. Experiments were performed on 600 simulated images as well as 20 image sequences obtained from 10 volunteer subjects, each sequence containing 50 images. Strategies for qualitative and quantitative evaluation consisted of visual assessment, feature overlap, similarity measures, and simulation experiments. The registration method has comparable performance to our previous registration technique; however, has the advantage of lower computational requirements and hence, is potentially more suitable for clinical applications. Rigorous performance evaluations attest to the fast speed of registration at an average of 5.5 frames per second on a conventional computing platform. (E-mail: purang@cs.queensu.ca)