Visual Search System Research Articles

CoReS: Compatible Representations via Stationarity.

Compatible features enable the direct comparison of old and new learned features allowing to use them interchangeably over time. In visual search systems, this eliminates the need to extract new features from the gallery-set when the representation model is upgraded with novel data. This has a big value in real applications as re-indexing the gallery-set can be computationally expensive when the gallery-set is large, or even infeasible due to privacy or other concerns of the application. In this paper, we propose CoReS, a new training procedure to learn representations that are compatible with those previously learned, grounding on the stationarity of the features as provided by fixed classifiers based on polytopes. With this solution, classes are maximally separated in the representation space and maintain their spatial configuration stationary as new classes are added, so that there is no need to learn any mappings between representations nor to impose pairwise training with the previously learned model. We demonstrate that our training procedure largely outperforms the current state of the art and is particularly effective in the case of multiple upgrades of the training-set, which is the typical case in real applications.

Neuromorphic computing for content-based image retrieval.

Neuromorphic computing mimics the neural activity of the brain through emulating spiking neural networks. In numerous machine learning tasks, neuromorphic chips are expected to provide superior solutions in terms of cost and power efficiency. Here, we explore the application of Loihi, a neuromorphic computing chip developed by Intel, for the computer vision task of image retrieval. We evaluated the functionalities and the performance metrics that are critical in content-based visual search and recommender systems using deep-learning embeddings. Our results show that the neuromorphic solution is about 2.5 times more energy-efficient compared with an ARM Cortex-A72 CPU and 12.5 times more energy-efficient compared with NVIDIA T4 GPU for inference by a lightweight convolutional neural network when batch size is 1 while maintaining the same level of matching accuracy. The study validates the potential of neuromorphic computing in low-power image retrieval, as a complementary paradigm to the existing von Neumann architectures.

Features of images of water, ice, snow, objects and a human formed by a hybrid television camera in the near-infrared range

Recently, television cameras operating in the near-infrared range have become increasingly widespread. The advantage of shooting in the short wave infrared range is the ability to observe objects in low light and difficult weather conditions. Such cameras can use hybrid sensors that consist of an infrared photocathode and an electron-sensitive matrix based on silicon in a single vacuum volume. The paper investigates the capabilities of one of the latest samples of a similar camera, created at JSC “NRI Electron”. The images of a human, water, ice, snow and other objects in the wavelength range of 0.95–1.7 microns have been analyzed. The images were taken using a television camera based on a hybrid television infrared sensor, which consists of a photocathode and a silicon charge-coupled device in proximity range to each other in a single vacuum volume. Illumination of the objects in the near-infrared range was produced by a continuous spectrum lamp with a maximum of the detected radiation at a wavelength of 1.55 μm. The authors compared the images obtained in the near-infrared range with the ones obtained in the visible region. An explanation is given for the differences between snow images and water and ice images in the near-infrared range. As an example, the difference in light transmission between the surfaces of materials for diving equipment, such as coated and open cell neoprene, is demonstrated. Due to a significant contrast shown in the near-infrared range by images of various objects on the surface of water and ice, it is possible to create an effective system for searching objects on the water. The paper discusses the advantages of the proposed visual search system compared to other systems, including passive systems and the ones operating in the MWIR and LWIR. The research outlined the prospects of using the new camera for building an effective search system for objects and people on water surfaces

Design of Visual Search and Positioning System Based on Labview + PLC

Aiming at the problem that the fixed lens cannot achieve full-area shooting recognition and obtaining the target position within the target area. This paper proposes using a moving vision lens to perform shooting recognition at fixed points and sub-regions within the set search path to determine the target object’s visual coordinates and use PID calculation to drive the actuator to complete the target workpiece grabbing. The system has high positioning accuracy through experimental verification, and the correct identification and capture can reach 100%.

Semantic Active Visual Search System Based on Text Information for Large and Unknown Environments.

Different high-level robotics tasks require the robot to manipulate or interact with objects that are in an unexplored part of the environment or not already in its field of view. Although much works rely on searching for objects based on their colour or 3D context, we argue that text information is a useful and functional visual cue to guide the search. In this paper, we study the problem of active visual search (AVS) in large unknown environments. In this paper, we present an AVS system that relies on semantic information inferred from texts found in the environment, which allows the robot to reduce the search costs by avoiding not promising regions for the target object. Our semantic planner reasons over the numbers detected from door signs to decide either perform a goal-directed exploration towards unknown parts of the environment or carefully search in the already known parts. We compared the performance of our semantic AVS system with two other search systems in four simulated environments. First, we developed a greedy search system that does not consider any semantic information, and second, we invited human participants to teleoperate the robot while performing the search. Our results from simulation and real-world experiments show that text is a promising source of information that provides different semantic cues for AVS systems.

The Cybernetic Return in Human Factors/Ergonomics (HFE)

The last few years have seen resurgence in interest within human factors/ergonomics (HFE) in cybernetics. HFE has a long association with cybernetics (e.g., the influence of signal detection and control theory on studies of vigilance, visual search and human-machine systems). The panel will discuss more recent applications of cybernetics and focus on the ‘messy complexity’ and dynamic properties of 21st Century systems and a variety of issues associated with the ‘cybernetic return’ in HFE, which include: the use of communication theory to probe deeper into how specific state of minds are formed, in this case deception and recent examples of ‘fake news’ (Hancock); the integration of artificial intelligence and systems and cognitive agents (Karwowski); the control of degrees of freedom in loosely coupled work systems (e.g., emergency care – Morineau); and team dynamics, performance and coordination in complex sociotechnical systems (Guastello, Wiltshire). The panel concludes with some reflections on the past, present and future of cybernetics within HFE (Drury).

Query Optimization of Data Based on Window Function and Distributed Cluster in Visual Academic Search System

Query Optimization of Data Based on Window Function and Distributed Cluster in Visual Academic Search System

Securing visual search queries in ubiquitous scenarios empowered by smart personal devices

Smart personal devices are assuming a fundamental role in the ubiquitous communication and computing arena. They provide new sophisticated cameras and new visual search interfaces and facilities that can drastically improve their presence and role in complex IoT-based critical infrastructures, such as healthcare monitoring and emergency systems, or remote access control facilities and smart authentication services. This new scenario calls for strong secure and resilient visual query mechanisms for these devices. In this work we propose an innovative secure visual search system, which is well-suited for ubiquitous computing scenarios empowered by modern smart personal devices. More precisely, we show how to insert, at the visual data acquisition time, a watermark inside the already compressed descriptor characterizing an MPEG-CDVS data stream used in visual queries, to make it possible to decode the watermark on the server side in order to improve the robustness against image-based identity spoofing. Such a security enforcement solution may be practical in several real-life applications involving visual queries performed from personal trusted devices, and it is particularly suitable in all those application domains that require performing visual queries with a high degree of security. It has been extensively tested and achieved satisfactory results: the presence of such a watermark does not affect the image matching performance and functionality.

Tackling Similarity Search for Soccer Match Analysis: Multimodal Distance Measure and Interactive Query Definition.

Analysts and coaches in soccer sports need to investigate large sets of past matches of opposing teams in a short time to prepare their teams for upcoming matches. Thus, they need appropriate methods and systems supporting them in searching for soccer moves for comparison and explanation. For the search of similar soccer moves, established distance and similarity measures typically only take spatiotemporal features like shape and speed of movement into account. However, movement in invasive team sports such as soccer includes much more than just a sequence of spatial locations. We propose an enhanced similarity measure integrating spatial, player, event as well as high level context such as pressure into the process of similarity search. We present a visual search system supporting analysts in interactively identifying similar contextual enhanced soccer moves in a dataset containing more than 60 soccer matches. Our approach is evaluated by several expert studies. The results of the evaluation reveal the large potential of enhanced similarity measures in the future.

Analyzing the Information Search Behavior and Intentions in Visual Information Systems

Visual information search systems support different search approaches such as targeted, exploratory or analytical search. Those visual systems deal with the challenge of composing optimal initial result visualization sets that face the search intention and respond to the search behavior of users. The diversity of these kinds of search tasks require different sets of visual layouts and functionalities, e.g. to filter, thrill-down or even analyze concrete data properties. This paper describes a new approach to calculate the probability towards the three mentioned search intentions, derived from users' behavior. The implementation is realized as a web-service, which is included in a visual environment that is designed to enable various search strategies based on heterogeneous data sources. In fact, based on an entered search query our developed search intention analysis web-service calculates the most probable search task, and our visualization system initially shows the optimal result set of visualizations to solve the task. The main contribution of this paper is a probability-based approach to derive the users' search intentions based on the search behavior enhanced by the application to a visual system.

Visual Recognition of Ancient Inscriptions Using Convolutional Neural Network and Fisher Vector

By bringing together the most prominent European institutions and archives in the field of Classical Latin and Greek epigraphy, the EAGLE project has collected the vast majority of the surviving Greco-Latin inscriptions into a single readily-searchable database. Text-based search engines are typically used to retrieve information about ancient inscriptions (or about other artifacts). These systems require that the users formulate a text query that contains information such as the place where the object was found or where it is currently located. Conversely, visual search systems can be used to provide information to users (like tourists and scholars) in a most intuitive and immediate way, just using an image as query. In this article, we provide a comparison of several approaches for visual recognizing ancient inscriptions. Our experiments, conducted on 17, 155 photos related to 14, 560 inscriptions, show that BoW and VLAD are outperformed by both Fisher Vector (FV) and Convolutional Neural Network (CNN) features. More interestingly, combining FV and CNN features into a single image representation allows achieving very high effectiveness by correctly recognizing the query inscription in more than 90% of the cases. Our results suggest that combinations of FV and CNN can be also exploited to effectively perform visual retrieval of other types of objects related to cultural heritage such as landmarks and monuments.

The Mobile Visual Search Guiding System Based on SIFT

In order to provide personal guiding service to visitors in the Picture Gallery, a mobile visual search guiding system based on SIFT(Scale Invariant Feature Transform) is proposed in this paper. The visitors can achieve the paintings’ information using the camera function of the mobile phones when they enjoy the famous paintings in the Picture Gallery. The mobile search visual system proposed in this paper can identify the paintings in the server side so there are no extra performance requirements for the mobile phones. Firstly, the mobile visual search system was introduced. Then the system was designed and realized. The classification training was based on SVM (Support Vector Machine). The SIFT feature extraction algorithm was used to enhance the recognition rate. After the image retrieval, the geometric consistency check was employed to choose the best matching image. Finally, the experimental results verified the feasibility and practicability of this system. Compared with the present guiding systems, this system can communicate with the visitors conveniently and offer abundant and all-around information to the visitors. In addition, using this guiding system, any auxiliary devices are not needed to be installed in the Picture Gallery.

Mobile multi-view object image search

High user interaction capability of mobile devices can help improve the accuracy of mobile visual search systems. At query time, it is possible to capture multiple views of an object from different viewing angles and at different scales with the mobile device camera to obtain richer information about the object compared to a single view and hence return more accurate results. Motivated by this, we propose a new multi-view visual query model on multi-view object image databases for mobile visual search. Multi-view images of objects acquired by the mobile clients are processed and local features are sent to a server, which combines the query image representations with early/late fusion methods and returns the query results. We performed a comprehensive analysis of early and late fusion approaches using various similarity functions, on an existing single view and a new multi-view object image database. The experimental results show that multi-view search provides significantly better retrieval accuracy compared to traditional single view search.

Topic modeling and improvement of image representation for large-scale image retrieval

In this paper, we present a new visual search system for finding similar images in a large database. However, there are a number of challenges regarding the robustness of the image representations and the efficiency of the retrieval framework. To tackle these challenges, we first propose an encoding technique based on soft-assignment of local features to convert an entire image into a single vector, which is a compact and discriminative representation. This encoded vector is suitable for most types of efficient indexing methods to produce an initial result. To compensate for the lack of incorporating geometric and object-related information during the encoding scheme, we then propose a probabilistic topic model to formalize the spatial structure among the local features. Moreover, the topic model allows us to effectively extract the object and background regions from the image. This is performed by a Markov Chain Monte Carlo algorithm for approximate inference. Finally, benefiting from the extracted objects in each image, we present a re-ranking scheme to automatically refine the initial search results. Our proposed retrieval framework has two major advantages: i) an aggregation strategy through soft-assignment improves the discriminative power of the representation, which has a determinative effect on the retrieval precision; and ii) the probabilistic latent topic model enables us to not only gain insight into the spatial structure of the image, but also handle a large variation in the object appearance. The experimental results from four benchmark datasets show that our approach provides competitive accuracy, and runs about ten times faster. Our studies also verify that proposed approach works effectively on large-scale databases of millions of images.

Learn to Rank Images: A Unified Probabilistic Hypergraph Model for Visual Search

In visual search systems, it is important to address the issue of how to leverage the rich contextual information in a visual computational model to build more robust visual search systems and to better satisfy the user’s need and intention. In this paper, we introduced a ranking model by understanding the complex relations within product visual and textual information in visual search systems. To understand their complex relations, we focused on using graph-based paradigms to model the relations among product images, product category labels, and product names and descriptions. We developed a unified probabilistic hypergraph ranking algorithm, which, modeling the correlations among product visual features and textual features, extensively enriches the description of the image. We conducted experiments on the proposed ranking algorithm on a dataset collected from a real e-commerce website. The results of our comparison demonstrate that our proposed algorithm extensively improves the retrieval performance over the visual distance based ranking.

A Hybrid Mobile Visual Search System With Compact Global Signatures

Mobile visual search systems typically compare a query image against a database of annotated images for accurate object recognition. On-server database matching can search a large database hosted in the cloud, but the query latency could suffer with slow network transmissions or server congestion . On-device database matching can ensure fast recognition responses regardless of network or server conditions , but a small amount of memory on the mobile device can severely limit the number of images that can be stored in an on- device database. This paper presents a new hybrid system that combines the advantages of on-device and on-server database matching. At the core of this system, we first develop a compact and discriminative global signature to characterize each image. Our global signature uses an optimized local feature count that is derived from a statistical analysis of the retrieval performance . We additionally create two extensions that exploit color information within images and relationships between similar database images to improve retrieval accuracy. Then, we propose methods for efficient interframe coding of a sequence of global signatures which are extracted from the viewfinder frames on the mobile device. A low bitrate stream of global signatures can be sent to the server at an uplink bitrate of less than 2 kbps to broaden the search range of the current query and to update the on-device database to help future queries.

Local consistent hierarchical Hough Match for image re-ranking

Geometric image re-ranking is a widely adopted phrase to refine the large-scale image retrieval systems built based upon popular paradigms such as Bag-of-Words (BoW) model. Its main idea can be treated as a sort of geometric verification targeting at reordering the initial returning list by previous similarity ranking metrics, e.g. Cosine distance over the BoW vectors between query image and reference ones. In the literature, to guarantee the re-ranking accuracy, most existing schemes requires the initial retrieval to be conducted by using a large vocabulary (codebook), corresponding to a high-dimensional BoW vector. However, in many emerging applications such as mobile visual search and massive-scale retrieval, the retrieval has to be conducted by using a compact BoW vector to accomplish the memory or time requirement. In these scenarios, the traditional re-ranking paradigms are questionable and new algorithms are urgently demanded. In this paper, we propose an accurate yet efficient image re-ranking algorithm specific for small vocabulary in aforementioned scenarios. Our idea is inspired by Hough Voting in the transformation space, where votes come from local feature matches. Most notably, this geometry re-ranking can easily been aggregated to the cutting-edge image based retrieval systems yielding superior performance with a small vocabulary and being able to store in mobile end facilitating mobile visual search systems. We further prove that its time complexity is linear in terms of the re-ranking instance, which is a significant advantage over the existing scheme. In terms of mean Average Precision, we show that its performance is comparable or in some cases better than the state-of-the-art re-ranking schemes.

Content-Based Visual Landmark Search via Multimodal Hypergraph Learning.

While content-based landmark image search has recently received a lot of attention and became a very active domain, it still remains a challenging problem. Among the various reasons, high diverse visual content is the most significant one. It is common that for the same landmark, images with a wide range of visual appearances can be found from different sources and different landmarks may share very similar sets of images. As a consequence, it is very hard to accurately estimate the similarities between the landmarks purely based on single type of visual feature. Moreover, the relationships between landmark images can be very complex and how to develop an effective modeling scheme to characterize the associations still remains an open question. Motivated by these concerns, we propose multimodal hypergraph (MMHG) to characterize the complex associations between landmark images. In MMHG, images are modeled as independent vertices and hyperedges contain several vertices corresponding to particular views. Multiple hypergraphs are firstly constructed independently based on different visual modalities to describe the hidden high-order relations from different aspects. Then, they are integrated together to involve discriminative information from heterogeneous sources. We also propose a novel content-based visual landmark search system based on MMHG to facilitate effective search. Distinguished from the existing approaches, we design a unified computational module to support query-specific combination weight learning. An extensive experiment study on a large-scale test collection demonstrates the effectiveness of our scheme over state-of-the-art approaches.

Locality Preserving Vector and Image-Specific Topic Model for Visual Recognition

Nowadays, evolution of mobile devices make demand for searching information increasing expressively. Many applications have been developed for recognition tasks. In this paper, we present a new and efficient visual search system for finding similar images on the large database. We first propose a compact, discriminative image representation called Locality Preserving Vector which can explicitly exploit neighborhood structure of data and attains high retrieval accuracy in the low-dimensional space. We then integrate topic modeling into visual search system for extracting topic related and image-specific information. These information enables images which likely contain the same objects to be ranked with higher similarity. The experiments show that our approach provides competitive accuracy with very low memory cost.

Memory-Efficient Image Databases for Mobile Visual Search

Mobile visual search systems compare images against a database for object recognition. If query data is transmitted over a slow network or processed on a congested server, the latency increases substantially. This article shows how on-device database matching guarantees fast recognition regardless of external conditions. The database signatures must be compact because of limited memory, capable of fast comparisons, and discriminative for robust recognition. The authors first describe methods that compress visual word histograms, which require a codebook and decoding compressed signatures. They then describe methods that use residuals to achieve the same accuracy with much smaller codebooks and compressed domain matching.