Compact Descriptors For Visual Search Research Articles

Mobile Visual Search Compression With Grassmann Manifold Embedding

With the increasing popularity of mobile phones and tablets, the explosive growth of query-by-capture applications calls for a compact representation of the query image feature. Compact descriptors for visual search (CDVS) is a recently released standard from the ISO/IEC moving pictures experts group, which achieves state-of-the-art performance in the context of image retrieval applications. However, they did not consider the matching characteristics in local space in a large-scale database, which might deteriorate the performance. In this paper, we propose a more compact representation with scale invariant feature transform (SIFT) descriptors for the visual query based on Grassmann manifold. Due to the drastic variations in image content, it is not sufficient to capture all the information using a single transform. To achieve more efficient representations, a SIFT manifold partition tree (SMPT) is initially constructed to divide the large dataset into small groups at multiple scales, which aims at capturing more discriminative information. Grassmann manifold is then applied to prune the SMPT and search for the most distinctive transforms. The experimental results demonstrate that the proposed framework achieves state-of-the-art performance on the standard benchmark CDVS dataset.

Codebook-Free Compact Descriptor for Scalable Visual Search

The MPEG compact descriptors for visual search (CDVS) is a standard toward image matching and retrieval. To achieve high retrieval accuracy over a large scale image/video dataset, recent research efforts have demonstrated that employing extremely high-dimensional descriptors such as the Fisher vector (FV) and the vector of locally aggregated descriptors (VLAD) can yield good performance. Since the FV (or VLAD) possesses high discriminability but small visual vocabulary, it has been adopted by CDVS to construct a global compact descriptor. In this paper, we study the development of global compact descriptors in the completed CDVS standard and the emerging compact descriptors for video analysis (CDVA) standard, in which we formulate the FV (or VLAD) compression as a resource-constrained optimization problem. Accordingly, we propose a codebook-free aggregation method via dual selection to generate a global compact visual descriptor, which supports fast and accurate feature matching free of large visual codebooks, fulfilling the low memory requirement of mobile visual search at significantly reduced latency. Specifically, we investigate both sample-specific Gaussian component redundancy and bit dependency within a binary aggregated descriptor to produce compact binary codes. Our technique contributes to the scalable compressed Fisher vector (SCFV) adopted by the CDVS standard. Moreover, the SCFV descriptor is currently serving as the frame-level hand-crafted video feature, which inspires the inheritance of CDVS descriptors for the emerging CDVA standard. Furthermore, we investigate the positive complementary effect of our standard compliant compact descriptor and deep learning based features extracted from convolutional neural networks with significant mean average precision gains. Extensive evaluation over benchmark databases shows the significant merits of the codebook-free binary codes for scalable visual search.

Feature Fusion for Image Retrieval With Adaptive Bitrate Allocation and Hard Negative Mining

By combining Convolutional Neural Network (CNN) descriptor and Compact Descriptors for Visual Search (CDVS), the visual search performance can be boosted. However, some redundancies still exist in the CDVS representation and the hard negative mining is not very accurate when training CNN embeddings. In this paper, we propose a high performance image retrieval scheme based on descriptor fusion. In detail, we first propose a more compact CDVS descriptor database building scheme through bitrate allocation, which can reduce information redundancy and boost image retrieval performance. We then propose a highly accurate CDVS-guided hard negative mining scheme when training CNN embeddings. In the hard negative selection, the CDVS descriptor and the CNN embedding are adaptively weighted together to achieve more precise decisions. Finally, the retrieval result is further refined through CDVS local descriptor matching by removing the irrelevant targets from the top positions. Extensive experimental results show that the proposed method outperforms the recent hybrid method and several other anchors remarkably, and produces better visual search performance. Codes and some models are available at https://github.com/WendyDong/ImageRetrieval_DF_CDVS .

Data-Driven Lightweight Interest Point Selection for Large-Scale Visual Search

With the explosive increase of images and videos, visual analysis has become an essential technique in dealing with the big visual data, which utilizes the visual feature descriptors to search or recognize the images or frames with target objects or events. Subject to the constraints of resources (e.g., memory, bandwidth, storage, etc.), interest point selection is crucial to generate robust compact descriptors for high-efficiency visual analysis by selecting and aggregating the most discriminative local feature descriptors, which has been demonstrated in the state-of-the-art low bit rate visual search works. In this paper, we propose a data-driven lightweight interest point selection approach to significantly improve the performance of visual search, while ameliorating the efficiency of extracting feature descriptors. Comprehensive experimental results over benchmarks have shown that the proposed interest point selection algorithm has significantly improved image matching and retrieval performance in the completed MPEG Compact Descriptors for Visual Search (CDVS) standard as well as the emerging MPEG Compact Descriptors for Video Analytics (CDVA) standard, say 20% mAP gain by data-driven selection against random selection of interest points. In particular, the presented data-driven interest point selection has been adopted by MPEG-CDVS and MPEG-CDVA as a normative technique to improve the aggregation of handcrafted features, which has contributed to the combination of handcrafted features and deep learning (CNN) features as well.

A Fast MPEG’s CDVS Implementation for GPU Featured in Mobile Devices

The Moving Picture Experts Group’s Compact Descriptors for Visual Search (MPEG’s CDVS) intends to standardize technologies in order to enable an interoperable, efficient, and cross-platform solution for internet-scale visual search applications and services. Among the key technologies within CDVS, we recall the format of visual descriptors, the descriptor extraction process, and the algorithms for indexing and matching. Unfortunately, these steps require precision and computation accuracy. Moreover, they are very time-consuming, as they need running times in the order of seconds when implemented on the central processing unit (CPU) of modern mobile devices. In this paper, to reduce computation times and maintain precision and accuracy, we re-design, for many-cores embedded graphical processor units (GPUs), all main local descriptor extraction pipeline phases of the MPEG’s CDVS standard. To reach this goal, we introduce new techniques to adapt the standard algorithm to parallel processing. Furthermore, to reduce memory accesses and efficiently distribute the kernel workload, we use new approaches to store and retrieve CDVS information on proper GPU data structures. We present a complete experimental analysis on a large and standard test set. Our experiments show that our GPU-based approach is remarkably faster than the CPU-based reference implementation of the standard, and it maintains a comparable precision in terms of true and false positive rates.

Mobile Visual Search Based on Histogram Matching and Zone Weight Learning

In this paper, we propose a novel image retrieval algorithm for mobile visual search. At first, a short visual codebook is generated based on the descriptor database to represent the statistical information of the dataset. Then, an accurate local descriptor similarity score is computed by merging the tf-idf weighted histogram matching and the weighting strategy in compact descriptors for visual search (CDVS). At last, both the global descriptor matching score and the local descriptor similarity score are summed up to rerank the retrieval results according to the learned zone weights. The results show that the proposed approach outperforms the state-of-the-art image retrieval method in CDVS.

Fast MPEG-CDVS Encoder With GPU-CPU Hybrid Computing.

The compact descriptors for visual search (CDVS) standard from ISO/IEC moving pictures experts group has succeeded in enabling the interoperability for efficient and effective image retrieval by standardizing the bitstream syntax of compact feature descriptors. However, the intensive computation of a CDVS encoder unfortunately hinders its widely deployment in industry for large-scale visual search. In this paper, we revisit the merits of low complexity design of CDVS core techniques and present a very fast CDVS encoder by leveraging the massive parallel execution resources of graphics processing unit (GPU). We elegantly shift the computation-intensive and parallel-friendly modules to the state-of-the-arts GPU platforms, in which the thread block allocation as well as the memory access mechanism are jointly optimized to eliminate performance loss. In addition, those operations with heavy data dependence are allocated to CPU for resolving the extra but non-necessary computation burden for GPU. Furthermore, we have demonstrated the proposed fast CDVS encoder can work well with those convolution neural network approaches which enables to leverage the advantages of GPU platforms harmoniously, and yield significant performance improvements. Comprehensive experimental results over benchmarks are evaluated, which has shown that the fast CDVS encoder using GPU-CPU hybrid computing is promising for scalable visual search.

관성 센서 데이터를 활용한 3 DoF 이미지 스티칭 향상

본 논문은 수평을 유지하여 촬영해야 한다는 기존 이미지 스티칭을 이용한 영상 정합 과정의 단점을 극복하기 위하여, 스마트폰의 가속도 센서와 자기장 센서 데이터를 사용하여 3가지 자유도(3 DoF)에 강인한 이미지 스티칭 방법을 제안한다. 이미지를 붙이는 작업인 이미지 스티칭은 크게 이미지 특징점 추출, 추출된 특징점에서 매칭에 필요한 참인 점(inlier)을 선별, 참인 점을 호모그래피(homography) 행렬로 변환, 호모그래피 행렬을 사용하여 이미지를 왜곡(warping), 왜곡된 이미지와 다른 이미지를 합하는 과정으로 이루어져 있다. 본 논문에서는 일반적으로 사용하는 SIFT, SURF 등의 알고리즘뿐만 아니라 MPEG에서 표준화한 MPEG-7 CDVS(Compact Descriptor for Visual Search) 표준의 특징점 추출 알고리즘을 사용하여 이미지의 특징점을 추출한다. 또한 각 알고리즘의 특징점 추출시간, 추출된 특징점 개수, 선별된 참인 점의 개수를 비교하고, 스티칭 정확도를 판단하여 본 연구에서 활용한 데이터에 어느 알고리즘이 효율적인지 살펴본다. This paper proposes a method to generate panoramic images by combining conventional feature extraction algorithms (e.g., SIFT, SURF, MPEG-7 CDVS) with sensed data from an inertia sensor to enhance the stitching results. The challenge of image stitching increases when the images are taken from two different mobile phones with no posture calibration. Using inertia sensor data obtained by the mobile phone, images with different yaw angles, pitch angles, roll angles are preprocessed and adjusted before performing stitching process. Performance of stitching (e.g., feature extraction time, inlier point numbers, stitching accuracy) between conventional feature extraction algorithms is reported along with the stitching performance with/without using the inertia sensor data.

Overview of the MPEG-CDVS Standard.

Compact descriptors for visual search (CDVS) is a recently completed standard from the ISO/IEC moving pictures experts group (MPEG). The primary goal of this standard is to provide a standardized bitstream syntax to enable interoperability in the context of image retrieval applications. Over the course of the standardization process, remarkable improvements were achieved in reducing the size of image feature data and in reducing the computation and memory footprint in the feature extraction process. This paper provides an overview of the technical features of the MPEG-CDVS standard and summarizes its evolution.

Feature selection for low bit rate mobile augmented reality applications

Mobile augmented reality applications rely on automatically matching a captured visual scene to an image in a database. This is typically achieved by deriving a set of features from the captured image, transmitting them through a network and then matching with features derived from a database of reference images. A fundamental problem is to select as few and robust features as possible such that the matching accuracy is invariant to distortions caused by camera capture whilst minimizing the bit rate required for their transmission. In this paper, novel feature selection methods are proposed, based on the entropy of the image content, entropy of extracted features and the Discrete Cosine Transformation (DCT) coefficients. The methods proposed in the descriptor domain and DCT domain achieve better retrieval accuracy under low bit rate transmission than state-of-the-art peak based feature selection used within the MPEG-7 Compact Descriptor for Visual Search (CDVS). The robustness of the proposed methods is evaluated under controlled single distortion and the retrieval performance is verified from image retrieval experiments and results for a realistic dataset with complex real world capturing distortion. Results show that the proposed method can improve the matching accuracy for various detectors. On the one side, feature selection can achieve low bit rate transmission, on the other side, when coping with distorted images, it can result in a higher matching accuracy than using all features. Hence, even if all the features can be transmitted in high transmission bandwidth scenarios, feature selection should still be applied to the distorted query image to ensure high matching accuracy.

Local detectors and compact descriptors for visual search: A quantitative comparison

Local Visual detectors and descriptors have been studied for many years, but their applications (e.g. mobile visual search) in large volume, low-cost, low-power embedded systems have been limited or negligible to date. One reason is the lack of a worldwide industry standard. MPEG Compact Descriptors for Visual Search (CDVS) working group filled this gap by defining a high-performance extraction stage and the bitstream syntax at its output in order to achieve interoperability between different implementations of clients and servers. In a previous work, we presented an analysis of various gray-level interest point detection and description algorithms, which was also contributed to CDVS. This work extends the previous analysis using the MPEG CDVS Test Model framework to consider additional detectors and the use of color descriptors.

Weighted Component Hashing of Binary Aggregated Descriptors for Fast Visual Search

Towards low bit rate mobile visual search, recent works have proposed to aggregate the local features and compress the aggregated descriptor (such as Fisher vector, the vector of locally aggregated descriptors) for low latency query delivery as well as moderate search complexity. Even though Hamming distance can be computed very fast, the computational cost of exhaustive linear search over the binary descriptors grows linearly with either the length of a binary descriptor or the number of database images. In this paper, we propose a novel weighted component hashing (WeCoHash) algorithm for long binary aggregated descriptors to significantly improve search efficiency over a large scale image database. Accordingly, the proposed WeCoHash has attempted to address two essential issues in Hashing algorithms: “what to hash” and “how to search.” “What to hash” is tackled by a hybrid approach, which utilizes both image-specific component (i.e., visual word) redundancy and bit dependency within each component of a binary aggregated descriptor to produce discriminative hash values for bucketing. “How to search” is tackled by an adaptive relevance weighting based on the statistics of hash values. Extensive comparison results have shown that WeCoHash is at least 20 times faster than linear search and 10 times faster than local sensitive hash (LSH) when maintaining comparable search accuracy. In particular , the WeCoHash solution has been adopted by the emerging MPEG compact descriptor for visual search (CDVS) standard to significantly speed up the exhaustive search of the binary aggregated descriptors.

Compact Descriptors for Visual Search

To ensure application interoperability in visual object search technologies, the MPEG Working Group has made great efforts in standardizing visual search technologies. Moreover, extraction and transmission of compact descriptors are valuable for next-generation, mobile, visual search applications. This article reviews the significant progress of MPEG Compact Descriptors for Visual Search (CDVS) in standardizing technologies that will enable efficient and interoperable design of visual search applications. In addition, the article presents the location search and recognition oriented data collection and benchmark under the MPEG CDVS evaluation framework.

Learning Compact Visual Descriptors for Low Bit Rate Mobile Landmark Search

Along with the ever‐growing computational power of mobile devices, mobile visual search has undergone an evolution in techniques and applications. A significant trend is low bit rate visual search, where compact visual descriptors are extracted directly over a mobile and delivered as queries rather than raw images to reduce the query transmission latency. In this article, we introduce our work on low bit rate mobile landmark search, in which a compact yet discriminative landmark image descriptor is extracted by using a location context such as GPS, crowd‐sourced hotspot WLAN, and cell tower locations. The compactness originates from the bag‐of‐words image representation, with offline learning from geotagged photos from online photosharing websites including Flickr and Panoramio. The learning process involves segmenting the landmark photo collection by discrete geographical regions using a Gaussian mixture model and then boosting a ranking‐sensitive vocabulary within each region, with “entropy”‐based feedback on the compactness of the descriptor to refine both phases iteratively. In online search, when entering a geographical region, the code book in a mobile device is downstream adapted to generate extremely compact descriptors with promising discriminative ability. We have deployed landmark search apps to both HTC and iPhone mobile phones, accessing a database of a million scale images in typical areas like Beijing, New York, and Barcelona, and others. Our descriptor outperforms alternative compact descriptors (Chen et al. 2009; Chen et al., 2010; Chandrasekhar et al. 2009a; Chandrasekhar et al. 2009b) by significant margins. Beyond landmark search, this article will summarize the MPEG standarization progress of compact descriptor for visual search (CDVS) (Yuri et al. 2010; Yuri et al. 2011) toward application interoperability.

영상 검색을 위한 점진적 블록 크기 기반의 효율적인 손실 좌표 압축 기술

Image retrieval research activity has moved its focus from global descriptors to local descriptors of feature point such as SIFT. MPEG is Currently working on standardization of effective coding of location and local descriptors of feature point in the context mobile based image search driven application in the name of MPEG-7 CDVS (Compact Descriptor for Visual Search). The extracted feature points consist of two parts, location information and Descriptor. For efficient image retrieval, we proposed a novel method that is gradual block-based efficient lossy location coding to compress location information according to distribution in images. From experimental result, the number of average bits per feature point reduce 5~6% and the accuracy rate keep compared to state of the art TM 3.0.

Laplacian embedding and key points topology verification for large scale mobile visual identification

Visual query-by-capture applications call for a compact visual descriptor with minimum descriptor length. Preserving the visual identification performance while minimising the bit rate is a focus of the on-going MPEG7 CDVS (Compact Descriptors for Visual Search) standardisation effort. In this paper we tackle this problem by adopting Laplacian embedding for SIFT feature compression and employing topology verification based on a novel graph cut measure. In contrast to previous feature compression schemes, we approach the problem by finding a Laplacian embedding that preserves the nearest neighbour relations in feature space. Furthermore, we develop an efficient yet effective topology verification (TV) scheme to perform spatial consistency checking. In contrast to previous works on geometric verification, instead of enumerating all possible combinations of coordinate alignments of an image pair, this TV solution verifies possibly misaligned coordinate sets with a learning method which acquires a proper boundary between the topology representation of matched and non-matched image pairs. Furthermore, this TV solution is invariant to in-plane rotation, scaling and is quite resilient to a range of out-of-plane rotations. The proposed Laplacian embedding and Topological verification scheme are tested with the CDVS dataset and are found to be effective.

Lossless Location Coding for Image Feature Points

Image retrieval research activity has moved its focus from global descriptors to local descriptors of feature point such as SIFT. Currently MPEG is working on standardization of effective coding of location and local descriptors of feature point in the context mobile based image search driven application in the name of MPEG-7 CDVS (Compact Descriptor for Visual Search). While CDVS is dealing with lossy compression of location information, this paper presents various lossless compression methods of locations and provides comparative experimental results for applications requiring fine matching precision. Among 4 methods presented in this paper, Experimental results show that Block-based lossless location coding using circular scan order method shows the best compression results with compression ratio achieving 2.5 to 1 with reference to Fixed-Length lossless location coding method.

최인접 거리 비율 정합을 이용한 영상 특징점 선택 방법

본 논문에서는 현재 진행 중인 MPEG(Motion Picture Experts Group, ISO/IEC JTC1 SC29 WG11)의 표준화 작업 중 CDVS(Compact Descriptor for Visual Search)의 CE-7(Core Experiment)인 특징점 선택에 대한 방법을 제안한다. 서술자의 경량화를 위해서는 영상으로부터 추출된 많은 수의 특징점들 중에서 영상 정합에 사용될 중요한 특징점들을 선택해야 한다. 본 논문에서는 최 인접 거리 비율 정합(Nearest Neighbor distance ratio matching) 방법에 의해 영상 정합 단계에서 사용되지 않고 버려지는 특징점들을 미리 추출 단에서 제거하는 방법 제안하였다. 제안된 방법을 통하여 적은 비트 전송률을 요하는 시스템에서 특징점의 낭비를 피할 수 있고 결과적으로 추가적인 특징점을 사용할 수 있으므로 전체적인 성능 향상을 얻을 수 있었다. 제안된 알고리즘을 통하여 Pair-wise 정합 실험에서 기존의 Test Model 대비 최고 2.3%의 성공율(True positive rate)의 향상을 보였다. In this paper, we propose a feature point selection method for MPEG CDVS CE-7 which is processing on International Standard task. Among a large number of extracted feature points, more important feature points which is used in image matching should be selected for the compactness of image descriptor. The proposed method is that remove the feature point in the extraction phase which is filtered by nearest neighbor distance ratio matching in the matching phase. We can avoid the waste of the feature point and employ additional feature points by the proposed method. The experimental results show that our proposed method can obtain true positive rate improvement about 2.3% in pair-wise matching test compared with Test Model.

Selection of local features for visual search

In a compact descriptor for visual search only a limited number of local features may be included. The estimated probability for correct match between keypoints provides a good criterion for selection of a subset.

Uniform query formalization in mobile visual search: From standards to practice

Mobile visual search is the task of finding information (visual or not) given an input picture or video usually taken by a hand-held device. Interest around this topic is being boosted by the increasing amount of digitally stored images and the widespread proliferation of camera enabled mobile devices, such as mobile phones, PDAs or tablets. This fact is increasing the urgency of novel solutions for challenging problems such as the efficient coding of compact visual descriptors and the interoperability of distributed visual search query interfaces. Currently, almost every visual search service offers a different retrieval interface and image metadata description format, preventing unified and efficient access. In this context, standardization groups such as ISO/IEC SC29/WG11 (MPEG) and ISO/IEC SC29/WG1 (JPEG) have been working to create unified interfaces for image repositories. In one hand, MPEG provides the ISO/IEC 15938-12 (MPEG Query Format, MPQF), which standardizes a query language for multimedia repositories and has also started an activity for standardizing compact descriptors for visual search (CDVS). On the other hand, JPEG is now finishing the ISO/IEC 24800 (JPSearch), which provides solutions to the image metadata interoperability problem. This paper analyzes how these standardization activities can be combined to satisfy the requirements posed by the mobile visual search scenario, which are their limitations and which would be the necessary actions to be taken by the standardization committees in order to overcome them.