Content-based Information Retrieval Research Articles

TH-iSSD: Design and Implementation of a Generic and Reconfigurable Near-Data Processing Framework

We present the design and implementation of TH-iSSD, a near-data processing framework to address the data movement problem. TH-iSSD does not pose any restriction to the hardware selection and is highly reconfigurable—its core components, such as the on-device compute unit (e.g., FPGA, embedded CPUs) and data collectors (e.g., camera, sensors), can be easily replaced to adapt to different use cases. TH-iSSD achieves this goal by incorporating highly flexible computation and data paths. In the data path, TH-iSSD adopts an efficient device-level data switch that exchanges data with both host CPUs and peripheral sensors; it also enables direct accesses between the sensing, computation, and storage hardware components, which completely eliminates the redundant data movement overhead, and thus delivers both high performance and energy efficiency. In the computation path, TH-iSSD provides an abstraction of filestream for developers, which abstracts a collection of data along with the related computation task as a file. Since existing applications are familiar with POSIX-like interfaces, they can be ported on top of our platform with minimal code modification. Moreover, TH-iSSD also introduces mechanisms including pipelined near-data processing and priority-aware I/O scheduling to make TH-iSSD perform more effectively. We deploy TH-iSSD to accelerate two types of applications: the content-based information retrieval system and the edge zero-streaming system. Our experimental results show that TH-iSSD achieves up to 1.6× higher throughput and 36% lower latency than compute-centric designs.

Human Behaviour Recognition using Fuzzy System in Videos

Human behavior can be detected and analyzed using video sequence is a latest research topic in computer vision & machine learning. Human behavior is used as a basis for many modern applications, such as video surveillance, content-based information retrieval from videos etc. HBA (Human behaviour analysis) is tricky to design and develop due to uncertainty and ambiguity involved in people’s daily activities. To address this gap, we propose hierarchical structure combining TDNN, tracking algorithms, and fuzzy systems. As a result, HBA system performance will be improved in terms of robustness, effectiveness and scalability.

The New Approach Research on Singing Voice Detection Algorithm Based on Enhanced Reconstruction Residual Network

With the development of Internet technology, multimedia information resources are increasing rapidly. Faced with the massive resources in the multimedia music library, it is extremely difficult for people to find the target music that meets their needs. How to realize computer analysis and perceive users’ needs for music resources has become the goal of the future development of human-computer interaction capabilities. Content-based music information retrieval applications are mainly embodied in the automatic classification and recognition of music. Traditional feedforward neural networks are prone to lose local information when extracting singing voice features. For this reason, on the basis of fully considering the impact of information persistence in the network propagation process, this paper proposes an enhanced two-stage super-resolution reconstruction residual network which can effectively integrate the learned features of each layer while increasing the depth of the network. The first stage of reconstruction is to complete the hierarchical learning of singing voice features through dense residual units to improve the integration of information. The second stage of reconstruction is mainly to perform residual relearning on the high-frequency information of the singing voice learned in the first stage to reduce the reconstruction error. In the middle of these two stages, the model introduces feature scaling and expansion convolution to achieve the dual purpose of reducing information redundancy and increasing the receptive field of the convolution kernel. A monophonic singing voice separation based on the high-resolution neural network is proposed. Because the high-resolution network has parallel subnetworks with different resolutions, it also has original resolution representations and multiple low-resolution representations, avoiding information loss caused by serial network downsampling effects and repeating multiple feature fusions to generate new semantic representations, allowing for the learning of comprehensive, high-precision, and highly abstract features. In this article, a high-resolution neural network is utilized to model the time spectrogram in order to correctly estimate the real value of the anticipated time-amplitude spectrograms. Experiments on the dataset MIR-1K show that compared with the current leading SH-4Stack model, the method in this paper has improved SDR, SIR, and SAR indicators for measuring the separation performance, confirming the effectiveness of the algorithm in this paper.

Techniques for Detecting Video Shot Boundaries: A Review

Due to the Corona Virus Diseases (COVID-19) pandemic, education is completely dependent on digital platforms, so recent advances in technology have made a tremendous amount of video content available. Due to the huge amount of video content, content-based information retrieval has become more and more important. Video content retrieval, just like information retrieval, requires some pre-processing such as indexing, key frame selection, and, most importantly, accurate detection of video shots. This gives the way for video information to be stored in a manner that will allow easy access. Video processing plays a vital role in many large applications. The applications required to perform the various manipulations on video streams (as on frames or say shots). The high definition of video can take a lot of memory to store, so compression techniques are huge in demand. Also, object tracking or object identification is an area where much considerable research has taken place and it is in progress.

Content Based Image and Video Retrieval: A Compressive Review

Content-based image retrieval is quickly becoming the most common method of searching vast databases for images, giving researchers a lot of room to develop new techniques and systems. Likewise, another common application in the field of computer vision is content-based visual information retrieval. For image and video retrieval, text-based search and Web-based image reranking have been the most common methods. Though Content Based Video Systems have improved in accuracy over time, they still fall short in interactive search. The use of these approaches has exposed shortcomings such as noisy data and inaccuracy, which often result in the showing of irrelevant images or videos. The authors of the proposed study integrate image and visual data to improve the precision of the retrieved results for both photographs and videos. In response to a user's query, this study investigates alternative ways for fetching high-quality photos and related videos.

Strategi Pengembangan Content Based Video Retrieval Di Perpustakaan Sekolah Tinggi Multi Media MMTC Yogyakarta

ABSTRACT
 
 Introduction One of the efforts to provide the best service for users is by developing innovative library services. One of them is by developing a video content-based library collection. MMTC Yogyakarta Multi Media College Library has developed a video content-based information retrieval system. It is hoped that by utilizing this video content-based STKI, users will be helped and get accelerated information in finding the material needed, especially searching for material in video files.
 Data Collection Method. In this paper the writer uses qualitative research with a library research approach, while the data analysis uses content analysis techniques. This method the authors use to observe and analyze an information system.
 Results and Discussions. In developing a Content Based Video Retrieval strategy in the MMTC Yogyakarta Multi Media High School Library, it begins with identifying user needs, creating a system design, evaluating the system design, pouring the system design into a programming language, testing the system, evaluating the system and using it. Then, the authors also provide an overview of the development of the STKI by conducting a SWOT analysis. Based on the macro analysis, the opportunity and threat variables will be formulated, while the internal analysis will formulate the strength and weakness variables. The last stage is the STKI analysis, while the stages are: complete definition, problem analysis, needs analysis, logic design and needs analysis.
 Conclusions. In the Content Based Video Retrieval development strategy at the MMTC Yogyakarta Multi Media College Library, there are several things that need to be considered in the development of an information retrieval system, including: User needs, development budget (budget), human resources, support from leaders and facilities (software and hardware) and IT infrastructure (internet network). The development of the STKI should begin with identifying user needs and conducting a SWOT analysis to determine the strengths and weaknesses of the system, as well as the goal so that the system can be optimally empowered by users.
 
 Keywords: Library, Information Retrieval System, Video Content

HerCulB: content-based information extraction and retrieval for cultural heritage of the Balkans

Purpose The purpose of this paper is to provide a methodology for automatic annotation of a multimedia collection of intangible cultural heritage mostly in the form of interviews. Assigned annotations provide a way to search the collection. Design/methodology/approach Annotation is based on automatic extraction of metadata and is conducted by named entity and topic extraction from textual descriptions with a rule-based approach supported by vocabulary resources, a compiled domain-specific classification scheme and domain-oriented corpus analysis. Findings The proposed methodology for automatic annotation of a collection of intangible cultural heritage, applied on the cultural heritage of the Balkans, has very good results according to F measure, which is 0.87 for the named entity and 0.90 for topic annotation. The overall methodology enables encapsulating domain-specific and language-specific knowledge into collections of finite state transducers and allows further improvements. Originality/value Although cultural heritage has a significant role in the development of identity of a group or an individual, it is one of those specific domains that have not yet been fully explored in case of many languages. A methodology is proposed that can be used for incorporating natural language processing techniques into digital libraries of cultural heritage.

Large-Scale Multi-modal Distance Metric Learning with Application to Content-Based Information Retrieval and Image Classification

Metric learning algorithms aim to make the conceptually related data items closer and keep dissimilar ones at a distance. The most common approach for metric learning on the Mahalanobis method. Despite its success, this method is limited to find a linear projection and also suffer from scalability respecting both the dimensionality and the size of input data. To address these problems, this paper presents a new scalable metric learning algorithm for multi-modal data. Our method learns an optimal metric for any feature set of the multi-modal data in an online fashion. We also combine the learned metrics with a novel Passive/Aggressive (PA)-based algorithm which results in a higher convergence rate compared to the state-of-the-art methods. To address scalability with respect to dimensionality, Dual Random Projection (DRP) is adopted in this paper. The present method is evaluated on some challenging machine vision datasets for image classification and Content-Based Information Retrieval (CBIR) tasks. The experimental results confirm that the proposed method significantly surpasses other state-of-the-art metric learning methods in most of these datasets in terms of both accuracy and efficiency.

Variance-preserving deep metric learning for content-based image retrieval

Supervised deep metric learning led to spectacular results for several Content-based Information Retrieval (CBIR) applications. The success of these approaches slowly led to the belief that image retrieval and classification are just slightly different variations of the same problem. However, recent evidence suggests that learning highly discriminative representation for a (limited) set of training classes removes valuable information from the representation, potentially harming both the in-domain, as well as the out-of-domain retrieval precision. In this paper, we propose a regularized discriminative deep metric learning method that aims to not only learn a representation that allows for discriminating between different classes, but it is also capable of encoding the latent generative factors separately for each class, overcoming this limitation. This allows for modeling the in-class variance and, as a result, maintaining the ability to represent both sub-classes of the in-domain data, as well as objects that belong to classes outside the training domain. The effectiveness of the proposed method, over existing supervised and unsupervised representation/metric learning approaches, is demonstrated under different in-domain and out-of-domain setups and three challenging image datasets.

A comparative study of data-dependent approaches without learning in measuring similarities of data objects

Conventional general-purpose distance-based similarity measures, such as Minkowski distance (also known as $$\ell _p$$-norm with $$p>0$$), are data-independent and sensitive to units or scales of measurement. There are existing general-purpose data-dependent measures, such as rank difference, Lin’s probabilistic measure and $$m_p$$-dissimilarity ($$p>0$$), which are not sensitive to units or scales of measurement. Although they have been shown to be more effective than the traditional distance measures, their characteristics and relative performances have not been investigated. In this paper, we study the characteristics and relationships of different general-purpose data-dependent measures. We generalise $$m_p$$-dissimilarity where $$p\ge 0$$ by introducing $$m_0$$-dissimilarity and show that it is a generic data-dependent measure with data-dependent self-similarity, of which rank difference and Lin’s measure are special cases with data-independent self-similarity. We evaluate the effectiveness of a wide range of general-purpose data-dependent and data-independent measures in the content-based information retrieval and kNN classification tasks. Our findings show that the fully data-dependent measure of $$m_p$$-dissimilarity is a more effective alternative to other data-dependent and commonly-used distance-based similarity measures as its task-specific performance is more consistent across a wide range of datasets.

Images Similarity based on Bags of SIFT Descriptor and K-Means Clustering

The content based image retrieval is developed and receives many attention from computer vision, supported by the ubiquity of Internet and digital devices. Bag-of-words method from text-based image retrieval trains images’ local features to build visual vocabulary. These visual words are used to represent local features, then quantized before clustering into number of bags. Here, the scale invariant feature transform descriptor is used as local features of images that will be compared each other to find their similarity. It is robust to clutter and partial visibility compared to global feature. The main goal of this research is to build and use a vocabolary to measure image similarity accross two tiny image datasets. K-means clustering algorithm is used to find the centroid of each cluster at different k values. From experiment results, the bag-of-keypoints method has potential to be implemented in the content based information retrieval.

Generative Model Based Video Shot Boundary Detection for Automated Surveillance

Video shot boundary detection (SBD) or video cut detection is one of the fundamental processes of video-processing with respect to semantic understanding, contextual information accumulation, labeling, content-based information retrieval and many more applications, such as video surveillance and monitoring. In this work, the authors have proposed a generative-model based framework for detecting shot boundaries in between the frames of a video segment. To generate a model of shot-boundaries, the authors have applied the concepts of Support Vector Machine to estimate the distance between any two images, and then, have generated a Gaussian Mixture Model from the estimated distances. Next, a Bayesian Estimation process checks the presence of boundaries in between the images by exploiting the Gaussian Mixture-based boundary model. Further, the authors have used the principles of Compressive Sensing to reduce the overhead of boundary detection process without losing of important information.

Generalized Residual Vector Quantization and Aggregating Tree for Large Scale Search

Vector quantization is an essential tool for tasks involving large scale data, for example, large scale similarity search, which is crucial for content-based information retrieval and analysis. In this paper, we propose a novel vector quantization framework that iteratively minimizes quantization error. First, we provide a detailed review on a relevant vector quantization method named residual vector quantization (RVQ). Next, we propose generalized residual vector quantization (GRVQ) to further improve over RVQ. Many vector quantization methods can be viewed as special cases of our proposed method. To enable GRVQ on billion scale data, we introduce a nonexhaustive search scheme named aggregating tree (A-Tree) for high dimensional data that uses GRVQ encodings to build a radix tree and perform the nearest neighbor search by beam search. To search accurately and efficiently, VQ-encodings should satisfy locally aggregating encoding criterion: For any node of the corresponding A-Tree, neighboring vectors should aggregate in fewer subtrees to make beam search efficient. We show that the proposed GRVQ encodings best satisfy the suggested criterion, and the joint use of GRVQ and A-Tree shows significantly better performances on billion scale datasets. Our methods are validated on several standard benchmark datasets. Experimental results and empirical analysis show the superior efficiency and effectiveness of our proposed methods compared to the state-of-the-art for large scale search.

Bucket-size balancing locality sensitive hashing using the map reduce paradigm

Similarity search is an essential operation in such domains as data mining and content-based information retrieval. This simple operation causes considerable burden when the number of data records grows large, especially in big data applications. At the sacrifice of accuracy, approximate methods for finding similar ones have been developed to deliver effective services in a reasonable amount of time. Locality sensitive hashing is a class of efficient approximate similarity search techniques. Various algorithms have been proposed for locality sensitive hashing, which basically try to narrow down the candidate data set to be examined. The candidate data set does not always contain all the similar data to query and thus the search results are approximate. The increase in the size of a candidate set improves the recall of similar ones, but it deteriorates the processing speed. This paper is concerned with a method to increase the recall rate while not entailing significant cost. The method basically uses a random hyperplane partitioning technique to create buckets to which data objects are distributed. The nearest neighbors located on the other side of such hyperplanes can be false negatives when only the bucket to which query belongs is examined for finding similar neighbors. The proposed method extends the hyperplanes to occupy their vicinity so that the data objects in the vicinity of a hyperplane are treated as belonging to both sides of the hyperplane simultaneously. The over-sized buckets are further split by adding additional hyperplanes to control the bucket sizes. To improve the processing speed, the algorithm is realized in MapReduce paradigm on a Hadoop cluster. Some experiment results are presented to show its applicability.

3D shape representation with spatial probabilistic distribution of intrinsic shape keypoints

The accelerated advancement in modeling, digitizing, and visualizing techniques for 3D shapes has led to an increasing amount of 3D models creation and usage, thanks to the 3D sensors which are readily available and easy to utilize. As a result, determining the similarity between 3D shapes has become consequential and is a fundamental task in shape-based recognition, retrieval, clustering, and classification. Several decades of research in Content-Based Information Retrieval (CBIR) has resulted in diverse techniques for 2D and 3D shape or object classification/retrieval and many benchmark data sets. In this article, a novel technique for 3D shape representation and object classification has been proposed based on analyses of spatial, geometric distributions of 3D keypoints. These distributions capture the intrinsic geometric structure of 3D objects. The result of the approach is a probability distribution function (PDF) produced from spatial disposition of 3D keypoints, keypoints which are stable on object surface and invariant to pose changes. Each class/instance of an object can be uniquely represented by a PDF. This shape representation is robust yet with a simple idea, easy to implement but fast enough to compute. Both Euclidean and topological space on object’s surface are considered to build the PDFs. Topology-based geodesic distances between keypoints exploit the non-planar surface properties of the object. The performance of the novel shape signature is tested with object classification accuracy. The classification efficacy of the new shape analysis method is evaluated on a new dataset acquired with a Time-of-Flight camera, and also, a comparative evaluation on a standard benchmark dataset with state-of-the-art methods is performed. Experimental results demonstrate superior classification performance of the new approach on RGB-D dataset and depth data.

Study of applications of content based information Retrieval models in healthcare

Study of applications of content based information Retrieval models in healthcare

Study of applications of content based information Retrieval models in healthcare

Study of applications of content based information Retrieval models in healthcare

A video summarization approach based on the emulation of bottom-up mechanisms of visual attention

This work addresses the development of a computational model of visual attention to perform the automatic summarization of digital videos from television archives. Although the television system represents one of the most fascinating media phenomena ever created, we still observe the absence of effective solutions for content-based information retrieval from video recordings of programs produced by this media universe. This fact relates to the high complexity of the content-based video retrieval problem, which involves several challenges, among which we may highlight the usual demand on video summaries to facilitate indexing, browsing and retrieval operations. To achieve this goal, we propose a new computational visual attention model, inspired on the human visual system and based on computer vision methods (face detection, motion estimation and saliency map computation), to estimate static video abstracts, that is, collections of salient images or key frames extracted from the original videos. Experimental results with videos from the Open Video Project show that our approach represents an effective solution to the problem of automatic video summarization, producing video summaries with similar quality to the ground-truth manually created by a group of 50 users.

Big Data for Remote Sensing: Challenges and Opportunities

Every day a large number of Earth observation (EO) spaceborne and airborne sensors from many different countries provide a massive amount of remotely sensed data. Those data are used for different applications, such as natural hazard monitoring, global climate change, urban planning, etc. The applications are data driven and mostly interdisciplinary. Based on this it can truly be stated that we are now living in the age of big remote sensing data. Furthermore, these data are becoming an economic asset and a new important resource in many applications. In this paper, we specifically analyze the challenges and opportunities that big data bring in the context of remote sensing applications. Our focus is to analyze what exactly does big data mean in remote sensing applications and how can big data provide added value in this context. Furthermore, this paper describes the most challenging issues in managing, processing, and efficient exploitation of big data for remote sensing problems. In order to illustrate the aforementioned aspects, two case studies discussing the use of big data in remote sensing are demonstrated. In the first test case, big data are used to automatically detect marine oil spills using a large archive of remote sensing data. In the second test case, content-based information retrieval is performed using high-performance computing (HPC) to extract information from a large database of remote sensing images, collected after the terrorist attack to the World Trade Center in New York City. Both cases are used to illustrate the significant challenges and opportunities brought by the use of big data in remote sensing applications.

Fast and reliable inference of semantic clusters

Document Indexing is but not limited to summarizing document contents with a small set of keywords or concepts of a knowledge base. Such a compact representation of document contents eases their use in numerous processes such as content-based information retrieval, corpus-mining and classification. An important effort has been devoted in recent years to (partly) automate semantic indexing, i.e. associating concepts to documents, leading to the availability of large corpora of semantically indexed documents. In this paper we introduce a method that hierarchically clusters documents based on their semantic indices while providing the proposed clusters with semantic labels. Our approach follows a neighbor joining strategy. Starting from a distance matrix reflecting the semantic similarity of documents, it iteratively selects the two closest clusters to merge them in a larger one. The similarity matrix is then updated. This is usually done by combining similarity of the two merged clusters, e.g. using the average similarity. We propose in this paper an alternative approach where the new cluster is first semantically annotated and the similarity matrix is then updated using the semantic similarity of this new annotation with those of the remaining clusters. The hierarchical clustering so obtained is a binary tree with branch lengths that convey semantic distances of clusters. It is then post-processed by using the branch lengths to keep only the most relevant clusters. Such a tool has numerous practical applications as it automates the organization of documents in meaningful clusters (e.g. papers indexed by MeSH terms, bookmarks or pictures indexed by WordNet) which is a tedious everyday task for many people. We assess the quality of the proposed methods using a specific benchmark of annotated clusters of bookmarks that were built manually. Each dataset of this benchmark has been clustered independently by several users. Remarkably, the clusters automatically built by our method are congruent with the clusters proposed by experts. All resources of this work, including source code, jar file, benchmark files and results are available at this address: http://sc.nicolasfiorini.info.