Visual Phrases Research Articles

Decoupled Cross-Modal Phrase-Attention Network for Image-Sentence Matching.

The mainstream of image and sentence matching studies currently focuses on fine-grained alignment of image regions and sentence words. However, these methods miss a crucial fact: the correspondence between images and sentences does not simply come from alignments between individual regions and words but from alignments between the phrases they form respectively. In this work, we propose a novel Decoupled Cross-modal Phrase-Attention network (DCPA) for image-sentence matching by modeling the relationships between textual phrases and visual phrases. Furthermore, we design a novel decoupled manner for training and inferencing, which is able to release the trade-off for bi-directional retrieval, where image-to-sentence matching is executed in textual semantic space and sentence-to-image matching is executed in visual semantic space. Extensive experimental results on Flickr30K and MS-COCO demonstrate that the proposed method outperforms state-of-the-art methods by a large margin, and can compete with some methods introducing external knowledge.

Disguising as I Go

Printmaking, Relief My body of work is a reflection of self. I am perceiving myself as a conscious being inside of a fleshy vessel and I explore the sentiments of that experience. I am building a relationship between myself and my perception of self. Striving to evoke malaise as a means to unearth buried emotions. I am investigating the transmutation of the internal self as it is expressed through the external self. Abstracted forms intended to reimagine the body’s physical reality are conceived under the pretense of stripping back the self. Internal dialogues are articulated and their narratives are emphasized through repetitive lines and fluid forms. As the rhythm accumulates, visual phrases are conceived in which the lines answer one another only to divulge into new conversations as they sprawl across the page. Objective and subjective reality come together to see beyond the discomfort and harmonize my mind and body.

An object detection and classification method for underwater visual images based on the bag-of-words model

In this study, an autonomous underwater vehicle (AUV) detection technique was used as the background to investigate the target classification method for underwater visual images. The target classification method is based on the bag-of-words model, with reduced classification accuracy when the edges of underwater objects are blurred, and when there are many noise points. To solve this problem, an improved visual dictionary generation method was proposed based on the bag-of-words model. The proposed method includes a step on decomposing the complete contour of the target before extracting the features. The DCE (discrete contour evolution) algorithm was used to decompose the complete contour into segments of different lengths, and the SC (shape context) method was applied to extract the features of the contour segments. The FCM (Fuzzy C-Means) algorithm was then used to generate a visual dictionary. To address the classification accuracy problem that causes the underwater targets to decrease after adding spatial location information to the bag-of-words model instead of increasing, an improved spatial location information description method was proposed. The contour segments with the nearest-neighbor relationship were represented as visual phrases, constituting the spatial information of the bag-of-words model. Additionally, for the problem of the shape information of contour segments with different shapes, and the relative position information between the contour segments having different contributions to the target classification, a method was developed to balance the contributions of the two using the optimal weight selected from the experiments. The experimental results demonstrated that the proposed method can effectively improve the classification accuracy of underwater target images.

Unambiguous Text Localization, Retrieval, and Recognition for Cluttered Scenes.

Text instance as one category of self-described objects provides valuable information for understanding and describing cluttered scenes. The rich and precise high-level semantics embodied in the text could drastically benefit the understanding of the world around us. While most recent visual phrase grounding approaches focus on general objects, this paper explores extracting designated texts and predicting unambiguous scene text information, i.e., to accurately localize and recognize a specific targeted text instance in a cluttered image from natural language descriptions (referring expressions). To address this issue, first a novel recurrent dense text localization network (DTLN) is proposed to sequentially decode the intermediate convolutional representations of a cluttered scene image into a set of distinct text instance detections. Our approach avoids repeated text detections at multiple scales by recurrently memorizing previous detections, and effectively tackles crowded text instances in close proximity. Second, we propose a context reasoning text retrieval (CRTR) model, which jointly encodes text instances and their context information through a recurrent network, and ranks localized text bounding boxes by a scoring function of context compatibility. Third, a recurrent text recognition module is introduced to extend the applicability of aforementioned DTLN and CRTR models, via text verification or transcription. Quantitative evaluations on standard scene text extraction benchmarks and a newly collected scene text retrieval dataset demonstrate the effectiveness and advantages of our models for the joint scene text localization, retrieval, and recognition task.

Unambiguous Scene Text Segmentation with Referring Expression Comprehension.

Text instance provides valuable information for the understanding and interpretation of natural scenes. The rich, precise high-level semantics embodied in the text could be beneficial for understanding the world around us, and empower a wide range of real-world applications. While most recent visual phrase grounding approaches focus on general objects, this paper explores extracting designated texts and predicting unambiguous scene text segmentation mask, i.e. scene text segmentation from natural language descriptions (referring expressions) like orange text on a little boy in black swinging a bat. The solution of this novel problem enables accurate segmentation of scene text instances from the complex background. In our proposed framework, a unified deep network jointly models visual and linguistic information by encoding both region-level and pixel-level visual features of natural scene images into spatial feature maps, and then decode them into saliency response map of text instances. To conduct quantitative evaluations, we establish a new scene text referring expression segmentation dataset: COCO-CharRef. Experimental results demonstrate the effectiveness of the proposed framework on the text instance segmentation task. By combining image-based visual features with language-based textual explanations, our framework outperforms baselines that are derived from state-of-the-art text localization and natural language object retrieval methods on COCO-CharRef dataset.

Image Retrieval using Visual Phrases

Keypoint based descriptors are widely used for various computer vision applications. During this process, key-points are initially detected from the given images which are later represented by some robust and distinctive descriptors like scale-invariant feature transform (SIFT). Keypoint based image-to-image matching has gained significant accuracy for image retrieval type of applications like image copy detection, similar image retrieval and near duplicate detection. Local keypoint descriptors are quantized into visual words to reduce the feature space which makes image-to-image matching possible for large scale applications. Bag of visual word quantization makes it efficient at the cost of accuracy. In this paper, the bag of visual word model is extended to detect frequent pair of visual words which is known as frequent item-set in text processing, also called visual phrases. Visual phrases increase the accuracy of image retrieval without increasing the vocabulary size. Experiments are carried out on benchmark datasets that depict the effectiveness of proposed scheme.

A robust CBIR framework in between bags of visual words and phrases models for specific image datasets

One objective of the Content Based Image Retrieval research field is to propose new methodologies and tools to manage the increasing number of images available. Linked to a specific context of small expert datasets without prior knowledge, our research work focuses on improving the discriminative power of the image representation while keeping the same efficiency for retrieval. Based on the well-known bag of visual words model, we propose three different methodologies inspired by the visual phrase model effectiveness and a compression technique which ensures the same effectiveness for retrieval than the BoVW model. Our experimental results study the performance of our proposals on different well known benchmark datasets and show its good performance compared to other recent approaches.

High discriminative SIFT feature and feature pair selection to improve the bag of visual words model

The bag of visual words (BOW) model has been widely applied in the field of image recognition and image classification. However, all scale-invariant feature transform (SIFT) features are clustered to construct the visual words which result in a substantial loss of discriminative power for the visual words. The corresponding visual phrases will further render the generated BOW histogram sparse. In this study, the authors aim to improve the classification accuracy by extracting high discriminative SIFT features and feature pairs. First, high discriminative SIFT features are extracted with the within- and between-class correlation coefficients. Second, the high discriminative SIFT feature pairs are selected by using minimum spanning tree and its total cost. Next, high discriminative SIFT features and feature pairs are exploited to construct the visual word dictionary and visual phrase dictionary, respectively, which are concatenated to a joint histogram with different weights. Compared with the state-of-the-art BOW-based methods, the experimental results on Caltech 101 dataset show that the proposed method has higher classification accuracy.

Image Location Inference by Multisaliency Enhancement

Locations of images have been widely used in many application scenarios for large geotagged image corpora. As to images that are not geographically tagged, we estimate their locations with the help of the large geotagged image set by content-based image retrieval. Bag-of-words image representation has been utilized widely. However, the individual visual word-based image retrieval approach is not effective in expressing the salient relationships of image region. In this paper, we present an image location estimation approach by multisaliency enhancement. We first extract region-of-interests (ROIs) by mean-shift clustering on the visual words and salient map of the image based on which we further determine the importance of the ROI. Then, we describe each ROI by the spatial descriptors of visual words. Finally, region-based visual phrases are generated to further enhance the saliency in image location estimation. Experiments show the effectiveness of our proposed approach.

Visual phrase recognition by modeling 3D spatial context of multiple objects

Automatically recognizing the visual phrase of an image is a challenging issue in computer vision. In this paper, we propose a method to discover and identify the visual phrase by automatically analyzing 3D spatial geometric structure of an image. It includes two steps: (1) learning 3D spatial geometric model; and (2) recognizing visual phrase. To achieve the first goal, we propose 3D geometric models (3DSG) that jointly capture both the features of objects and 3D spatial layout among objects in a visual phrase. In the second step, we transform the visual phrase recognition into verification by measuring the similarity of spatial configuration between the given visual pattern and the 3DSG model. The nature of our method makes itself precisely determine whether the given visual pattern belongs to a specific 3DSG model or not by maximizing the joint probability of the given visual pattern and a 3DSG model. Experiments conducted on several datasets show that our model outperforms the state-of-the-art models in modeling 3D spatial geometric structure as well as recognizing visual phrase. The results also demonstrate that modeling 3D spatial configuration between objects can significantly improve the deeper image understanding.

Improving retrieval framework using information gain models

Content-based image retrieval systems are meant to retrieve the most similar images of a collection to a query image. One of the most well-known models widely applied for this task is the bag of visual words (BoVW) model. In this paper, we introduce a study of different information gain models used for the construction of a visual vocabulary. In the proposed framework, information gain models are used as a discriminative information to index image features and select the ones that have the highest information gain values. We introduce some extensions to further improve the performance of the proposed framework: mixing different vocabularies and extending the BoVW to bag of visual phrases. Exhaustive experiments show the interest of information gain models on our retrieval framework.

Mining Visual Phrases for Visual Robot Localization

We propose a discriminative and compact scene descriptor for single-view place recognition that facilitates long-term visual SLAM in familiar, semi-dynamic, and partially changing environments. In contrast to popular bag-of-words scene descriptors, which rely on a library of vector quantized visual features, our proposed scene descriptor is based on a library of raw image data (such as an available visual experience, images shared by other colleague robots, and publicly available image data on the Web) and directly mine it to find visual phrases (VPs) that discriminatively and compactly explain an input query/database image. Our mining approach is motivated by recent success achieved in the field of common pattern discovery – specifically mining of common visual patterns among scenes – and requires only a single library of raw images that can be acquired at different times or on different days. Experimental results show that, although our scene descriptor is significantly more compact than conventional descriptors, its recognition performance is relatively high.

From Visual Attributes to Adjectives through Decompositional Distributional Semantics

As automated image analysis progresses, there is increasing interest in richer linguistic annotation of pictures, with attributes of objects (e.g., furry, brown…) attracting most attention. By building on the recent “zero-shot learning” approach, and paying attention to the linguistic nature of attributes as noun modifiers, and specifically adjectives, we show that it is possible to tag images with attribute-denoting adjectives even when no training data containing the relevant annotation are available. Our approach relies on two key observations. First, objects can be seen as bundles of attributes, typically expressed as adjectival modifiers (a dog is something furry, brown, etc.), and thus a function trained to map visual representations of objects to nominal labels can implicitly learn to map attributes to adjectives. Second, objects and attributes come together in pictures (the same thing is a dog and it is brown). We can thus achieve better attribute (and object) label retrieval by treating images as “visual phrases”, and decomposing their linguistic representation into an attribute-denoting adjective and an object-denoting noun. Our approach performs comparably to a method exploiting manual attribute annotation, it out-performs various competitive alternatives in both attribute and object annotation, and it automatically constructs attribute-centric representations that significantly improve performance in supervised object recognition.

Bayes pooling of visual phrases for object retrieval

Object retrieval is still an open question. A promising approach is based on the matching of visual phrases. However, this routine is often corrupted by visual phrase burstiness, i.e., the repetitive occurrence of some certain visual phrases. Burstiness leads to over-counting the co-occurring visual patterns between two images, thus would deteriorate the accuracy of image similarity measurement. On the other hand, existing methods are incapable of capturing the complete geometric variation between images. In this paper, we propose a novel strategy to address the two problems. Firstly, we propose a unified framework for matching geometry-constrained visual phrases. This framework provides a possibility of combing the optimal geometry constraints to improve the validity of matched visual phrases. Secondly, we propose to address the problem of visual phrase burstiness from a probabilistic view. This approach effectively filters out the bursty visual phrases through explicitly modelling their distribution. Experiments on five benchmark datasets demonstrate that our method outperforms other approaches consistently and significantly.

Extending Laplacian sparse coding by the incorporation of the image spatial context

Diverse studies have shown the efficiency of sparse coding in feature quantization. However, its major drawback is that it neglects the relationships among features. To reach the spatial context, we proposed in this paper, a novel sparse coding method called Extended Laplacian Sparse Coding. Two successive stages are required in this method. In the first stage, the sparse visual phrases based on Laplacian sparse coding are generated from the local regions in order to represent the geometric information in the image space. The second stage aims to incorporate the spatial relationships among local features in the image space into the objective function of the Laplacian sparse coding. It takes into account the similarity among local regions in the Laplacian sparse coding process. The matching between the local regions is based on the Hungarian method as well as the histogram intersection measure between sparse visual phrases already assigned to the local regions in the first stage. Furthermore, we suggested to improve the pooling step that succeeds the encoding step by introducing the discretized max pooling method that estimates the distribution of the responses of each local feature to the dictionary of basis vectors. Our experimental results prove that our method outperforms the existing background results.

Histogram of dense subgraphs for image representation

Modelling spatial information of local features is known to improve performance in image categorisation. Compared with simple pairwise features and visual phrases, graphs can capture the structural organisation of local features more adequately. Besides, a dense regular grid can guarantee a more reliable representation than the interest points and give better results for image classification. In this study, the authors introduced a bag of dense local graphs approach that combines the performance of bag of visual words expressing the image classification process with the representational power of graphs. The images were represented with dense local graphs built upon dense scale-invariant feature transform descriptors. The graph-based substructure pattern mining algorithm was applied on the local graphs to discover the frequent local subgraphs, producing a bag of subgraphs representation. The results were reported from experiments conducted on four challenging benchmarks. The findings show that the proposed subgraph histogram improves the categorisation accuracy.

Unsupervised mining of visually consistent shots for sports genre categorization over large-scale database

In this paper, an algorithm is proposed to summarize sports videos based on viewpoints in TV broadcasts for sports genre classification. The redundancy of multiple views is one of the principal limitations in sports genre classification. In order to remove the redundancy, the algorithm chooses the most representative subset of shots from each game. After videos are broken into shots, single keyframe is utilized to represent each shot and uniform LBP feature is extracted to represent each keyframe. Agglomerative hierarchical clustering is then performed to cluster these keyframes. In this step, an energy-based function for clusters is introduced to match the statistical distribution of various views, and a refined distance metric is proposed as similarity measure of two shots. We modify the energy function to meet the fact that temporally neighbored shots with similar duration are more likely to be in the same views. To make full use of the high overlap of selected key-frames subset, sparse coding and geometry visual phrase are introduced in the sports genre categorization part. Our method is evaluated on videos recorded from Orangesports, ESPN and Eurosport TV broadcast. The average accuracy over 10 sports reaches 87.5 %. The proposed algorithm is already applied in the Orange TV video content delinearization service platform.

Spatial Pooling of Heterogeneous Features for Image Classification

In image classification tasks, one of the most successful algorithms is the bag-of-features (BoFs) model. Although the BoF model has many advantages, such as simplicity, generality, and scalability, it still suffers from several drawbacks, including the limited semantic description of local descriptors, lack of robust structures upon single visual words, and missing of efficient spatial weighting. To overcome these shortcomings, various techniques have been proposed, such as extracting multiple descriptors, spatial context modeling, and interest region detection. Though they have been proven to improve the BoF model to some extent, there still lacks a coherent scheme to integrate each individual module together. To address the problems above, we propose a novel framework with spatial pooling of complementary features. Our model expands the traditional BoF model on three aspects. First, we propose a new scheme for combining texture and edge-based local features together at the descriptor extraction level. Next, we build geometric visual phrases to model spatial context upon complementary features for midlevel image representation. Finally, based on a smoothed edgemap, a simple and effective spatial weighting scheme is performed to capture the image saliency. We test the proposed framework on several benchmark data sets for image classification. The extensive results show the superior performance of our algorithm over the state-of-the-art methods.

Multi-order visual phrase for scalable partial-duplicate visual search

Visual phrase considers multiple visual words and captures extra spatial clues among them. Thus, visual phrase shows better discriminative power than single visual word in image retrieval and matching. Not withstanding their success, existing visual phrases still show obvious shortcomings: (1) limited flexibility, i.e., visual phrases are considered for matching only if they contain the same number of visual words; (2) large quantization error and low repeatability, i.e., quantization errors in visual words are aggregated in visual word combinations and visual phrases, making them harder to be matched than single visual words. To avoid these issues, we propose multi-order visual phrase (MVP) which contains two complementary clues: center visual word quantized from the local descriptor of each image keypoint and the visual and spatial clues of multiple nearby keypoints. Two MVPs are flexibly matched by first matching their center visual words, then estimating a match confidence by checking the spatial and visual consistency of their neighbor keypoints. Therefore, center visual word matching equals to traditional visual word matching, but the neighbor spatial and visual clues checking significantly boosts the discriminative power. MVP does not scarify the repeatability of single visual word and is more robust to quantization error than existing visual phrases. We test our approach in three image retrieval tasks on UKbench, Oxford5K, and 1 million distractor images collected from Flickr. Comparisons with recent retrieval approaches and existing visual phrase features clearly demonstrate the competitive accuracy and significantly better efficiency of MVP.

Discriminative Soft Bag-of-Visual Phrase for Mobile Landmark Recognition

This paper proposes a new bag-of-visual phrase (BoP) approach for mobile landmark recognition based on discriminative learning of category-dependent visual phrases. Many previous landmark recognition works adopt a bag-of-words (BoW) method which ignores the co-occurrence relationship between neighboring visual words in an image. Although some works that focus on visual phrase learning have appeared, they mainly construct a generalized phrase dictionary from all categories for recognition, which lacks descriptive capability for a specific category. Another shortcoming of these works is the hard assignment of numerous feature sets to a limited number of phrases, which causes some useful feature sets to be discarded, and yields information loss. In view of this, this paper presents a discriminative soft BoP approach for mobile landmark recognition. The candidate phrases defined as adjacent pairwise codewords are first generated for each category. The important candidates are then selected through a proposed discriminative visual phrase (DVP) selection approach to form the BoP dictionary. Finally, a soft encoding method is developed to quantize each image into a BoP histogram. The context information such as location and direction captured by mobile devices is also integrated with the proposed BoP-based content analysis for landmark recognition. Experimental results on two datasets show that the proposed method is effective in mobile landmark recognition.