Mobile Image Retrieval Research Articles

OMCBIR: Offline mobile content-based image retrieval with lightweight CNN optimization

Convolutional Neural Networks (CNNs) have achieved great success in computer vision applications. However, due to the high requirements for computation power and memory usage, most state-of-the-art CNNs are difficult to deploy on resource-constrained mobile devices. Although many typical lightweight neural networks have been proposed in the industry, such as MobileNetV2, which reduce the amount of parameters and calculations, they still have a lot of redundancy. Furthermore, few papers consider the use of deep learning models to implement image retrieval on terminals, so we propose a new offline retrieval framework based on lightweight neural network models, called Offline Mobile Content-Based Image Retrieval (OMCBIR). In this framework, we focus on the feature extraction of the model, by introducing pointwise group convolution and channel shuffle into the bottleneck block, reconstructing the network structure, and introducing the convolutional attention module, we propose an extremely lightweight small network-Attention-based Lightweight Network (ALNet). Compared to MobileNetV2, ALNet obtains a higher mAP on each dataset in OMCBIR when the model parameters are reduced by more than 62% and the model size is reduced by more than 63%. Extensive experiments conducted on five public datasets provide a trade-off between retrieval performance and model size of different algorithms, which proves the efficiency of the proposed OMCBIR.

Cooperative Cloud-Edge Feature Extraction Architecture for Mobile Image Retrieval

Mobile image retrieval greatly facilitates our lives and works by providing various retrieval services. The existing mobile image retrieval scheme is based on mobile cloud-edge computing architecture. That is, user equipment captures images and uploads the captured image data to the edge server. After preprocessing these captured image data and extracting features from these image data, the edge server uploads the extracted features to the cloud server. However, the feature extraction on the cloud server is noncooperative with the feature extraction on the edge server which cannot extract features effectively and has a lower image retrieval accuracy. For this, we propose a collaborative cloud-edge feature extraction architecture for mobile image retrieval. The cloud server generates the projection matrix from the image data set with a feature extraction algorithm, and the edge server extracts the feature from the uploaded image with the projection matrix. That is, the cloud server guides the edge server to perform feature extraction. This architecture can effectively extract the image data on the edge server, reduce network load, and save bandwidth. The experimental results indicate that this scheme can upload few features to get high retrieval accuracy and reduce the feature matching time by about 69.5% with similar retrieval accuracy.

A Cloud-Guided Feature Extraction Approach for Image Retrieval in Mobile Edge Computing

Mobile Edge Computing (MEC) can facilitate various important image retrieval applications for mobile users by offloading partial computation tasks from resource-limited mobile devices to edge servers. However, existing related works suffer from two major limitations. (i) High network bandwidth cost : they need to extract numerous features from the image and upload these feature data to the cloud server. (ii) Low retrieval accuracy : they separate the feature extraction processes from the image data set in the cloud server, thus unable to provide effective features for accurate image retrieval. In this paper, we propose a cloud-guided feature extraction approach for mobile image retrieval. In the proposed approach, the cloud server first leverages the relationships among labeled images in the data set to learn a projection matrix ${{\mathbf P}}$ P . Then, it uses the matrix ${{\mathbf P}}$ P to extract discriminative features from the image data set and form a low-dimensional feature data set. Following that, the cloud server sends the matrix ${{\mathbf P}}$ P to the edge server and uses it to multiply the image ${{\mathbf x}}$ x . The result ${{\mathbf P}}^T{{\mathbf x}}$ P T x , i.e. , image features, is uploaded to the cloud server to find the label of the image with the most similar multiplying result. The label is regarded as the retrieval result and returned to the mobile user. In the cloud-guided feature extraction approach, the matrix ${{\mathbf P}}$ P can extract a small number of effective image features, which not only reduces network traffic but also improves retrieval accuracy. We have implemented a prototype system to validate the proposed approach and evaluate its performance by conducting extensive experiments using a real MEC environment and data set. The experimental results show that the proposed approach reduces the network traffic by nearly 93 percent and improves the retrieval accuracy by nearly 6.9 percent compared with the state-of-the-art image retrieval approaches in MEC.

Unsupervised Topic Hypergraph Hashing for Efficient Mobile Image Retrieval.

Hashing compresses high-dimensional features into compact binary codes. It is one of the promising techniques to support efficient mobile image retrieval, due to its low data transmission cost and fast retrieval response. However, most of existing hashing strategies simply rely on low-level features. Thus, they may generate hashing codes with limited discriminative capability. Moreover, many of them fail to exploit complex and high-order semantic correlations that inherently exist among images. Motivated by these observations, we propose a novel unsupervised hashing scheme, called topic hypergraph hashing (THH), to address the limitations. THH effectively mitigates the semantic shortage of hashing codes by exploiting auxiliary texts around images. In our method, relations between images and semantic topics are first discovered via robust collective non-negative matrix factorization. Afterwards, a unified topic hypergraph, where images and topics are represented with independent vertices and hyperedges, respectively, is constructed to model inherent high-order semantic correlations of images. Finally, hashing codes and functions are learned by simultaneously enforcing semantic consistence and preserving the discovered semantic relations. Experiments on publicly available datasets demonstrate that THH can achieve superior performance compared with several state-of-the-art methods, and it is more suitable for mobile image retrieval.

Scalable mobile image retrieval by exploring contextual saliency.

Nowadays, it is very convenient to capture photos by a smart phone. As using, the smart phone is a convenient way to share what users experienced anytime and anywhere through social networks, it is very possible that we capture multiple photos to make sure the content is well photographed. In this paper, an effective scalable mobile image retrieval approach is proposed by exploring contextual salient information for the input query image. Our goal is to explore the high-level semantic information of an image by finding the contextual saliency from multiple relevant photos rather than solely using the input image. Thus, the proposed mobile image retrieval approach first determines the relevant photos according to visual similarity, then mines salient features by exploring contextual saliency from multiple relevant images, and finally determines contributions of salient features for scalable retrieval. Compared with the existing mobile-based image retrieval approaches, our approach requires less bandwidth and has better retrieval performance. We can carry out retrieval with <200-B data, which is <5% of existing approaches. Most importantly, when the bandwidth is limited, we can rank the transmitted features according to their contributions to retrieval. Experimental results show the effectiveness of the proposed approach.

Learning salient visual word for scalable mobile image retrieval

Owing to the portable and excellent phone camera, people now prefer to take photos and share them in social networks with their friends. If a user wants to obtain relevant information about an image, content based image retrieval method can be utilized. Taking the limited bandwidth and instability of wireless channel into account, in this paper we propose an effective scalable mobile image retrieval approach by exploiting the advantage of mobile end that people usually take multiple photos of an object in different viewpoints and focuses. The proposed algorithm first determines the truly relevant photos according to visual similarity in mobile end, then learns salient visual words by exploring saliency from these relevant images, and finally determines the contribution order of salient visual words to carry out scalable retrieval. Moreover, to improve the retrieval performance, soft spatial verification is proposed to re-rank the results. Compared to the existing approaches of mobile image retrieval, our approach transmits less data and reduces the computational cost of spatial verification. Most importantly, when the bandwidth is limited, we can transmit only a part of features according their contributions to retrieval. Experimental results show the effectiveness of the proposed approach.

모바일 플랫폼에서 다중 특징 기반의 이미지 검색

본 논문에서는 다양한 검색 환경과 모바일 디바이스의 센서 정보를 활용한 모바일 이미지 검색 방법을 제안하고 안드로이드 플랫폼에서 구동하는 검색 시스템을 구현하였다. 설계 개발 시스템은 JPEG 이미지를 대상으로 산업계 표준 메타데이터인 EXIF 속성과 시각적 특징을 결합한 새로운 검색 기술자이며, 검색을 위한 특징 추출 및 유사도 평가 알고리즘을 모바일 환경에 최적화한 이미지 검색 모듈이다. 실험을 통해, 대용량 이미지 데이터셋을 대상으로 안드로이드 폰에서 효율적인 이미지 검색을 수행하였음을 보였다. In this paper, we propose a mobile image retrieval system which utilizes the mobile device's sensor information and enables running in a variety of the environments, and implement the system on Android platform. The proposed system deals with a new image descriptor using combination of the visual feature with EXIF attributes in the target of JPEG image, and image matching algorithm which is optimized to the mobile environments. Experiments are performed on the Android platform, and the experimental results revealed that the proposed algorithm exhibits a significant improved results with large image database.

Content-based image retrieval using spatial-color and Gabor texture on a mobile device

Mobile image retrieval is one of the most exciting and fastest growing research fields in the area of multimedia technology. As the amount of digital contents continues to grow users are experiencing increasing difficulty in finding specific images in their image libraries. This paper proposes a new efficient and effective mobile image retrieval method that applies a weighted combination of color and texture utilizing spatial-color and second order statistics. The system for mobile image searches runs in real-time on an iPhone and can easily be used to find a specific image. To evaluate the performance of the new method, we assessed the Xcode simulations performance in terms of average precision and F-score using several image databases and compare the results with those obtained using existing methods such as MPEG-7. Experimental trials revealed that the proposed descriptor exhibited a significant improvement of over 13% in retrieval effectiveness, compared to the best of the other descriptors.

Searching for images by video

Image retrieval based on the query-by-example (QBE) principle is still not reliable enough, largely because of the likely variations in the capture conditions (e.g. light, blur, scale, occlusion) and viewpoint between the query image and the images in the collection. In this paper, we propose a framework in which this problem is explicitly addressed to improve the reliability of QBE-based image retrieval. We aim at the use scenario involving the user capturing the query object by his/her mobile device and requesting information augmenting the query from the database. Reliability improvement is achieved by allowing the user to submit not a single image but a short video clip as a query. Since a video clip may combine object or scene appearances captured from different viewpoints and under different conditions, the rich information contained therein can be exploited to discover the proper query representation and to improve the relevance of the retrieved results. The experimental results show that video-based image retrieval (VBIR) is significantly more reliable than the retrieval using a single image as query. Furthermore, to make the proposed framework deployable in a practical mobile image retrieval system, where realtime query response is required, we also propose the priority queue-based feature description scheme and cache-based bi-quantization algorithm for an efficient parallel implementation of the VBIR concept.

JPSearch: An answer to the lack of standardization in mobile image retrieval

The amount of imagery that is available via various content providers grows at a staggering rate. Contemporary handheld devices allow accessing these content providers anytime, anyplace. Moreover, using the build-in camera, one can easily contribute to these online image repositories. The present lack of consistency in how repositories are accessed for retrieval or syncing complicates inter-operability between devices and systems. JPSearch, a recent standardization initiative within the Joint Photographic Experts Group (JPEG) committee, resolves this predicament by defining the interfaces and protocols for data exchange between them, while restricting as little as possible how those devices, systems or components perform their task.

P-SURF: A Robust Local Image Descriptor

SIFT-like representations are considered as being most resistant to common deformations, although their computational burden is heavy for low-computation applications such as mobile image retrieval. H. Bay et al. proposed an efficient implementation ofSIFT called SURF. Although this descriptor has been able to represent the nature of some underlying image patterns, it is not enough to represent more complicated ones. Also, the proposed high-dimensional alternative to SURF indeed improves the distinctive character of the descriptor, while it appears to be less robust. In this paper, an enhanced version of SURF is proposed. Specifically, it consists of two components: the feature representation for independent intensity changes and the coupling description for these intensity changes. To this end, phase space is introduced to model the relationships between the intensitychanges and several statistic metrics quantizing these relationships are also proposed to meet practical demands. The feature matching experiments demonstrate that our method achieves a favorable performance close to that of SIFT and faster construction-speed. We also present results showing that the use of the enhanced SURF representation in a mobileimage retrieval application results in a comparable performance to SIFT.

Compressed Histogram of Gradients: A Low-Bitrate Descriptor

Establishing visual correspondences is an essential component of many computer vision problems, which is often done with local feature-descriptors. Transmission and storage of these descriptors are of critical importance in the context of mobile visual search applications. We propose a framework for computing low bit-rate feature descriptors with a 20× reduction in bit rate compared to state-of-the-art descriptors. The framework offers low complexity and has significant speed-up in the matching stage. We show how to efficiently compute distances between descriptors in the compressed domain eliminating the need for decoding. We perform a comprehensive performance comparison with SIFT, SURF, BRIEF, MPEG-7 image signatures and other low bit-rate descriptors and show that our proposed CHoG descriptor outperforms existing schemes significantly over a wide range of bitrates. We implement the descriptor in a mobile image retrieval system and for a database of 1 million CD, DVD and book covers, we achieve 96% retrieval accuracy using only 4 KB of data per query image.

PPD: A Robust Low-computation Local Descriptor for Mobile Image Retrieval

This paper proposes an efficient and yet powerful local descriptor called phase-space partition based descriptor (PPD). This descriptor is designed for the mobile image matching and retrieval. PPD, which is inspired from SIFT, also encodes the salient aspects of the image gradient in the neighborhood around an interest point. However, without employing SIFT’s smoothed gradient orientation histogram, we apply the region based gradient statistics in phase space to the construction of a feature representation, which allows to reduce much computation requirements. The feature matching experiments demonstrate that PPD achieves favorable performance close to that of SIFT and faster building and matching. We also present results showing that the use of PPD descriptors in a mobile image retrieval application results in a comparable performance to SIFT.