Content-based Retrieval Method Research Articles

Cross-Modal Retrieval-enhanced code Summarization based on joint learning for retrieval and generation

Context:Code summarization refers to a task that automatically generates a natural language description of a code snippet to facilitate code comprehension. Existing methods have achieved satisfactory results by incorporating information retrieval into generative deep-learning models for reusing summaries of existing code. However, most of these existing methods employed non-learnable generic retrieval methods for content-based retrieval, resulting in a lack of diversity in the retrieved results during training, thereby making the model over-reliant on retrieved results and reducing the generative model’s ability to generalize to unknown samples. Objective:To address this issue, this paper introduces CMR-Sum: a novel Cross-Modal Retrieval-enhanced code Summarization framework based on joint learning for generation and retrieval tasks, where both two tasks are allowed to be optimized simultaneously. Method:Specifically, we use a cross-modal retrieval module to dynamically alter retrieval results during training, which enhances the diversity of the retrieved results and maintains a relative balance between the two tasks. Furthermore, in the summary generation phase, we employ a cross-attention mechanism to generate code summaries based on the alignment between retrieved and generated summaries. We conducted experiments on three real-world datasets, comparing the performance of our method with baseline models. Additionally, we performed extensive qualitative analysis. Result:Results from qualitative and quantitative experiments indicate that our approach effectively enhances the performance of code summarization. Our method outperforms both the generation-based and the retrieval-enhanced baselines. Further ablation experiments demonstrate the effectiveness of each component of our method. Results from sensitivity analysis experiments suggest that our approach achieves good performance without requiring extensive hyper-parameter search. Conclusion:The direction of utilizing retrieval-enhanced generation tasks shows great potential. It is essential to increase the diversity of retrieval results during the training process, which is crucial for improving the generality and the performance of the model.

A Symbolic Contour Representation of Shape for Recognition and Retrieval

Image retrieval technique is one of the most important and vital part of huge image storages. A precise retrieval engine is required to meet the needs of users. In this regard, content-based image retrieval (CBIR) methods based on image contents such as shape, color and texture have been introduced as an alternative of traditional methods to improve the retrieval precision. Of all features of an image, shape holds more valuable information than others. However, due to the dynamic nature of real worlds’ objects, a shape of an object is subject to occlusion, transformation and deformation. This is the main challenge of shape-based retrieval techniques and shape descriptors. In this paper, a new shape representation and description called Symbolic Contour-based Shape Retrieval (SCSR) method is introduced to overcome drawbacks of previous methods in terms of computation time and accuracy. The performance of the proposed method has been compared to other state-of-art methods on three popular datasets, Vehicles, Brown and Mpeg7.

Privacy-Preserving Content-Based Image Retrieval Using Compressible Encrypted Images

In this article, we propose a novel content-based image-retrieval (CBIR) scheme using compressible encrypted images called “encryption-then-compression (EtC) images.” The proposed scheme allows us not only to directly retrieve images from visually protected images but to also apply EtC images that can be compressed by using the JPEG standard for the first time. In addition, the sensitive management of secret keys is not required in our framework. The proposed retrieval scheme is carried out on the basis of weighted searching images with MPEG-7-powered localized descriptors (weighted SIMPLE descriptors) combining scalable color descriptor (SCD) or color and edge directivity descriptor (CEDD). Weighted SIMPLE descriptors are extended, and CEDD is also modified to avoid the influence of image encryption. In an experiment, the proposed scheme is demonstrated to have almost no degradation in retrieval performance compared with conventional content-based retrieval methods with plain images under the use of two datasets. In addition, the proposed scheme is shown to outperform conventional privacy-preserving CBIR schemes including state-of-the-art ones in terms of mean average precision (mAP) scores.

Video retrieval for shot cluster and classification based on key feature set

ABSTRACTThe proposed content-based video retrieval method in this study uses feature extraction, shot boundary detection, shot noise filter, shot cluster, the key feature set of shot extraction and shot classification. Shot boundary detection is conducted using a weight selection genetic algorithm. In order to verify the video retrieval method in this paper, three video databases were used in the experiment. These video databases are used to verify the shot boundary detection, the shot classification and video retrieval of this study. A series of comparisons and analyses were then conducted using the above video databases. The experiment results showed that the shot boundary detection method exhibits better performance than the other compared methods. Moreover, the video retrieval experiment results showed that the proposed method is able to precisely catch a shot in the search video, and do so with reduced video retrieval time.

Retrieval in Long-Surveillance Videos Using User-Described Motion and Object Attributes

We present a content-based retrieval method for long-surveillance videos in wide-area (airborne) and near-field [closed-circuit television (CCTV)] imagery. Our goal is to retrieve video segments, with a focus on detecting objects moving on routes, that match user-defined events of interest. The sheer size and remote locations where surveillance videos are acquired necessitates highly compressed representations that are also meaningful for supporting user-defined queries. To address these challenges, we archive long-surveillance video through lightweight processing based on low-level local spatiotemporal extraction of motion and object 2. These are then hashed into an inverted index using locality-sensitive hashing. This local approach allows for query flexibility and leads to significant gains in compression. Our second task is to extract partial matches to user-created queries and assemble them into full matches using dynamic programming (DP). DP assembles the indexed low-level features into a video segment that matches the query route by exploiting causality. We examine CCTV and airborne footage, whose low contrast makes motion extraction more difficult. We generate robust motion estimates for airborne data using a tracklets generation algorithm, while we use the Horn and Schunck approach to generate motion estimates for CCTV. Our approach handles long routes, low contrasts, and occlusion. We derive bounds on the rate of false positives and demonstrate the effectiveness of the approach for counting, motion pattern recognition, and abandoned object applications.

A novel 3D model retrieval system based on three-view sketches

© 2016 - IOS Press and the authors. All rights reserved.3D models can be used in 3D printing and many other areas. At present, there are a lot of researches on 3D model retrieval and sketch is considered to be important for 3D model retrieval. In this paper, we develop a new 3D model retrieval prototype system based on style-sensitive 3D model retrieval method and three-view user sketches. We also implement user-friendly graphic interfaces for the 3D model retrieval system. We explore the performance of the system by conducting a series of 3D model retrieval experiments on the Princeton shape benchmark data set. Experimental results show that the new retrieval system can obtain satisfactory retrieval results. And some results of the new method are superior to some content-based 3D model retrieval methods, in terms of both quantitative search performance metrics and qualitatively measured user search experiences.

Modelling-based experiment retrieval: a case study with gene expression clustering.

Public and private repositories of experimental data are growing to sizes that require dedicated methods for finding relevant data. To improve on the state of the art of keyword searches from annotations, methods for content-based retrieval have been proposed. In the context of gene expression experiments, most methods retrieve gene expression profiles, requiring each experiment to be expressed as a single profile, typically of case versus control. A more general, recently suggested alternative is to retrieve experiments whose models are good for modelling the query dataset. However, for very noisy and high-dimensional query data, this retrieval criterion turns out to be very noisy as well. We propose doing retrieval using a denoised model of the query dataset, instead of the original noisy dataset itself. To this end, we introduce a general probabilistic framework, where each experiment is modelled separately and the retrieval is done by finding related models. For retrieval of gene expression experiments, we use a probabilistic model called product partition model, which induces a clustering of genes that show similar expression patterns across a number of samples. The suggested metric for retrieval using clusterings is the normalized information distance. Empirical results finally suggest that inference for the full probabilistic model can be approximated with good performance using computationally faster heuristic clustering approaches (e.g. k-means). The method is highly scalable and straightforward to apply to construct a general-purpose gene expression experiment retrieval method. The method can be implemented using standard clustering algorithms and normalized information distance, available in many statistical software packages. paul.blomstedt@aalto.fi or samuel.kaski@aalto.fi Supplementary data are available at Bioinformatics online.

The effects of multiple query evidences on social image retrieval

System performance assessment and comparison are fundamental for large-scale image search engine development. This article documents a set of comprehensive empirical studies to explore the effects of multiple query evidences on large-scale social image search. The search performance based on the social tags, different kinds of visual features and their combinations are systematically studied and analyzed. To quantify the visual query complexity, a novel quantitative metric is proposed and applied to assess the influences of different visual queries based on their complexity levels. Besides, we also study the effects of automatic text query expansion with social tags using a pseudo relevance feedback method on the retrieval performance. Our analysis of experimental results shows a few key research findings: (1) social tag-based retrieval methods can achieve much better results than content-based retrieval methods; (2) a combination of textual and visual features can significantly and consistently improve the search performance; (3) the complexity of image queries has a strong correlation with retrieval results’ quality—more complex queries lead to poorer search effectiveness; and (4) query expansion based on social tags frequently causes search topic drift and consequently leads to performance degradation.

Research of Music Retrieval System Based on Emotional Music Template

Traditional music retrieval system based on text information description can't meet people's demand for intelligent retrieval, on which basis content-based music retrieval method came into being. Emotional needs are introduced into retrieval and related researches are done to music retrieval method based on the emotion. This paper first constructs music emotion space to obtain the user's emotions; and then proposes emotional music template library through the study of the definition of emotional music model to meet users emotional needs matching template; Finally, based on this, advances the music retrieval system model based on emotional music template, trying to explore a kind of effective retrieval method based on emotion.

A Fair and Efficient Agent Scheduling Method for Content-Based Information Retrieval with Individual Time Constraints and Its Implementation

A Fair and Efficient Agent Scheduling Method for Content-Based Information Retrieval with Individual Time Constraints and Its Implementation

Multimodal biomedical image indexing and retrieval using descriptive text and global feature mapping

The images found within biomedical articles are sources of essential information useful for a variety of tasks. Due to the rapid growth of biomedical knowledge, image retrieval systems are increasingly becoming necessary tools for quickly accessing the most relevant images from the literature for a given information need. Unfortunately, article text can be a poor substitute for image content, limiting the effectiveness of existing text-based retrieval methods. Additionally, the use of visual similarity by content-based retrieval methods as the sole indicator of image relevance is problematic since the importance of an image can depend on its context rather than its appearance. For biomedical image retrieval, multimodal approaches are often desirable. We describe in this work a practical multimodal solution for indexing and retrieving the images contained in biomedical articles. Recognizing the importance of text in determining image relevance, our method combines a predominately text-based image representation with a limited amount of visual information, in the form of quantized content-based visual features, through a process called global feature mapping. The resulting multimodal image surrogates are easily indexed and searched using existing text-based retrieval systems. Our experimental results demonstrate that our multimodal strategy significantly improves upon the retrieval accuracy of existing approaches. In addition, unlike many retrieval methods that utilize content-based visual features, the response time of our approach is negligible, making it suitable for use with large collections.

Combining color and shape descriptors for 3D model retrieval

Nowadays many three dimensional models feature color information together with the shape description. However current content-based retrieval schemes for 3D models are based on shape information only and ignore color clues. The significance of shape vs. color clues for 3D model retrieval is instead a fundamental issue still almost unexplored at this time. A possible approach is to extend shape-based 3D model retrieval methods of proven effectiveness in order to include color. This work follows such rationale and introduces an extended version of the spin-image descriptor that can account also for color data. The comparison of color descriptors is performed using a novel scheme that allows to recognize as similar also objects with different colors but with the same color distribution over the shape. Shape and color similarity are finally combined together by an algorithm based on fuzzy logic. Experimental results show how the joint use of color and shape data allows to obtain better results than each of the two types of information alone. Comparisons with state-of-the-art content-based retrieval methods for 3D models also show how the proposed scheme outperforms standard solutions on object classes with meaningful color information.

Adaptive photograph retrieval method

Access to electronic books, electronic journals, and web portals, which may contain graphics (drawings or diagrams) and images, is now ubiquitous. However, users may have photographs that contain graphics or images and want to access an electronic database to retrieve this information. Hence, an effective photograph retrieval method is needed. Although many content-based retrieval methods have been developed for images and graphics, few are designed to retrieve graphics and images simultaneously. Moreover, existing graphics retrieval methods use contour-based rather than pixel-based approaches. Contour-based methods, which are concerned with lines or curves, are inappropriate for images. To retrieve graphics and images simultaneously, this work applies an adaptive retrieval method. The proposed method uses histograms of oriented gradient (HOG) as pixel-based features. However, the characteristics of graphics and images differ, and this affects feature extraction and retrieval accuracy. Thus, an adaptive method is proposed that selects different HOG-based features for retrieving graphics and images. Experimental results demonstrate the proposed method has high retrieval accuracy even under noisy conditions.

Content-Based Analysis Improves Audiovisual Archive Retrieval

Content-based video retrieval is maturing to the point where it can be used in real-world retrieval practices. One such practice is the audiovisual archive, whose users increasingly require fine-grained access to broadcast television content. In this paper, we take into account the information needs and retrieval data already present in the audiovisual archive, and demonstrate that retrieval performance can be significantly improved when content-based methods are applied to search. To the best of our knowledge, this is the first time that the practice of an audiovisual archive has been taken into account for quantitative retrieval evaluation. To arrive at our main result, we propose an evaluation methodology tailored to the specific needs and circumstances of the audiovisual archive, which are typically missed by existing evaluation initiatives. We utilize logged searches, content purchases, session information, and simulators to create realistic query sets and relevance judgments. To reflect the retrieval practice of both the archive and the video retrieval community as closely as possible, our experiments with three video search engines incorporate archive-created catalog entries as well as state-of-the-art multimedia content analysis results. A detailed query-level analysis indicates that individual content-based retrieval methods such as transcript-based retrieval and concept-based retrieval yield approximately equal performance gains. When combined, we find that content-based video retrieval incorporated into the archive's practice results in significant performance increases for shot retrieval and for retrieving entire television programs. The time has come for audiovisual archives to start accommodating content-based video retrieval methods into their daily practice.

Content-Based Retrieval of Focal Liver Lesions Using Bag-of-Visual-Words Representations of Single- and Multiphase Contrast-Enhanced CT Images

This paper is aimed at developing and evaluating a content-based retrieval method for contrast-enhanced liver computed tomographic (CT) images using bag-of-visual-words (BoW) representations of single and multiple phases. The BoW histograms are extracted using the raw intensity as local patch descriptor for each enhance phase by densely sampling the image patches within the liver lesion regions. The distance metric learning algorithms are employed to obtain the semantic similarity on the Hellinger kernel feature map of the BoW histograms. The different visual vocabularies for BoW and learned distance metrics are evaluated in a contrast-enhanced CT image dataset comprised of 189 patients with three types of focal liver lesions, including 87 hepatomas, 62 cysts, and 60 hemangiomas. For each single enhance phase, the mean of average precision (mAP) of BoW representations for retrieval can reach above 90 % which is significantly higher than that of intensity histogram and Gabor filters. Furthermore, the combined BoW representations of the three enhance phases can improve mAP to 94.5 %. These preliminary results demonstrate that the BoW representation is effective and feasible for retrieval of liver lesions in contrast-enhanced CT images.

Intelligent copyright protection system using a matching video retrieval algorithm

Many video service sites headed by YouTube know what content requires copyright protection. However, they lack a copyright protection system that automatically distinguishes whether uploaded videos contain legal or illegal content. Existing protection techniques use content-based retrieval methods that compare the features of video. However, if the video encoding has changed in resolution, bit-rate or codec, these techniques do not perform well. Thus, this paper proposes a novel video matching algorithm even if the type of encoding has changed. We also suggest an intelligent copyright protection system using the proposed algorithm. This can serve to automatically prevent the uploading of illegal content. The proposed method has represented the accuracy of 97% with searching algorithm in video-matching experiments and 98.62% with automation algorithm in copyright-protection experiments. Therefore, this system could form a core technology that identifies illegal content and automatically excludes access to illegal content by many video service sites.

Film Analysis of Archived Documentaries

This article outlines issues related to automated film analysis for archived documentaries and explores examples of the applicability of existing content-based retrieval methods.

PeerLearning: A Content-Based e-Learning Material Sharing System Based on P2P Network

Managing and retrieving reusable learning materials in a content-based way is a big challenge in e-Learning material sharing systems. E-Learning materials are highly heterogeneous; they may exist in the form of video, audio, image, slide or plain text. Furthermore, the learning systems are highly dynamic in the presence of massively increasing multimedia materials. P2P network seems to be one of the most promising infrastructures to deal with the challenge in such highly dynamic environments. In this paper we propose a Peer-to-Peer (P2P) infrastructure based on the trie tree and the deBruijn structure. It can support efficiently query processing in highly dynamic scenarios. Furthermore we develop a P2P e-Learning system PeerLearning to provide two content-based learning material sharing services: a keyword search component for supporting content-based document sharing and a content-based retrieval method for multimedia materials. Extensive experiments are conducted in this study to verify the superiority of our methods over the existing works.

Relevance ranking in georeferenced video search

The rapid adoption and deployment of ubiquitous video cameras has led to the collection of voluminous amounts of media data. However, indexing and searching of large video databases remain a very challenging task. Recently, some recorded video data are automatically annotated with meta-data collected from various sensors such as Global Positioning System (GPS) and compass devices. In our earlier work, we proposed the notion of a viewable scene model derived from the fusion of location and direction sensor information with a video stream. Such georeferenced media streams are useful in many applications and, very importantly, they can effectively be searched via their meta-data on a large scale. Consequently, search by geo-properties complements traditional content-based retrieval methods. The result of a georeferenced video query will in general consist of a number of video segments that satisfy the query conditions, but with more or less relevance. For example, a building of interest may appear in a video segment, but may only be visible in a corner. Therefore, an essential and integral part of a video query is the ranking of the result set according to the relevance of each clip. An effective result ranking is even more important for video than it is for text search, since the browsing of results can only be achieved by viewing each clip, which is very time consuming. In this study, we investigate and present three ranking algorithms that use spatial and temporal properties of georeferenced videos to effectively rank search results. To allow our techniques to scale to large video databases, we further introduce a histogram-based approach that allows fast online computations. An experimental evaluation demonstrates the utility of the proposed methods.

An efficient high-dimensional indexing method for content-based retrieval in large image databases

High-dimensional indexing methods have been proved quite useful for response time improvement. Based on Euclidian distance, many of them have been proposed for applications where data vectors are high-dimensional. However, these methods do not generally support efficiently similarity search when dealing with heterogeneous data vectors. In this paper, we propose a high-dimensional indexing method (KRA +-Blocks) as an extension of the region approximation approach to the kernel space. KRA +-Blocks combines nonlinear dimensionality reduction technique (KPCA) with region approximation approach to map data vectors into a reduced feature space. The created feature space is then used, on one hand to approximate regions, and on the other hand to provide an effective kernel distances for both filtering process and similarity measurement. In this way, the proposed approach achieves high performances in response time and in precision when dealing with high-dimensional and heterogeneous vectors.