Visual Content Descriptors Research Articles

A Novel Method for Feature Extraction using Color Layout Descriptor (CLD) and Edge histogram Descriptor (EHD)

Now days, Image processing finds diversified applications in almost all field of life. The success of any image processing application is depends on proper feature extraction technique. To extract good and proper features is very interesting and challenging task in the development process. It is used to describe the image based on its contents. These extracted features are used to compare, analyse and/or search the analogous images. There are various feature extraction techniques are found in the literature to design various applications. However any image processing application generates images with high dimensionality, which will be results in the low efficiency of an application. This paper provides an approach to extract features from the images using MPEG-7 feature extraction techniques. The approach discussed in the paper uses two popular MPEG-7 visual content descriptors; they are namely Edge Histogram Descriptor (EHD) and Color Layout Descriptor (CLD). The concept results in reduction of dimensions of an image to improve the efficiency of the application. It can be used as a heart to design any image processing application as well as provides strong foundation to develop variety of applications.

Compact Hash Codes for Efficient Visual Descriptors Retrieval in Large Scale Databases

In this paper, we present an efficient method for visual descriptors retrieval based on compact hash codes computed using a multiple k-means assignment. The method has been applied to the problem of approximate nearest neighbor (ANN) search of local and global visual content descriptors, and it has been tested on different datasets: three large scale standard datasets of engineered features of up to one billion descriptors (BIGANN) and, supported by recent progress in convolutional neural networks (CNNs), on CIFAR-10, MNIST, INRIA Holidays, Oxford 5K, and Paris 6K datasets; also, the recent DEEP1B dataset, composed by one billion CNN-based features, has been used. Experimental results show that, despite its simplicity, the proposed method obtains a very high performance that makes it superior to more complex state-of-the-art methods.

Optimized Image Retrieval System in Oracle DBMS

In this work, the authors are moving towards the creation of an effective image retrieval system in Oracle DBMS. Several DBMSs have been extensively used to manage the textual information stored with images and CBIR tasks usually rely on specific applications. The separation between the DBMSs and CBIR prevents the optimization of integrated search process based on the connection between the textual and visual content description of image. Moreover, the relevance of image retrieval depends directly on the choice of similarity criteria (color, texture, shape) that can give inaccurate results in case of non-trivial selection of these parameters. The purpose of the authors' approach is to build a CBIR system using advanced and integrated retrieval techniques defined in Oracle DBMS. This approach provides an assistance tool that can guide the user to the appropriate choice of search criteria. The authors present an experimental part that measures the performance of their system, which can help the user to correctly model his query by giving the appropriate retrieval criteria for a database with 800 images.

Audio Description in Interactive Television (iTV): Proposal of a Collaborative and Voluntary Approach

Audio description (AD) is a media feature especially designed for visually impaired people to enable the access, via audio, to the description of visual content, such as television programs or movies, in which a narrator orally describes the visual scenes that are not perceptible by these users. Without this feature, unfortunately not available on a large scale, visually impaired users (VIU) have major difficulties in taking the best benefit of television contents.Currently, the production of AD is a costly and complex process, achieving only the minimum requirements mandated by what TV law obliges. However, the current technological convergence paradigm provides great opportunities to enhance this process, leveraging the number of AD and bringing relevant socio-economic impacts.In this context, this paper aims to describe some key steps to support the participatory development of a cross-platform prototype, which enable collaborative and voluntary creation of AD in an inclusive approach. A special focus is given to a preliminary test aiming to understand how the target audience responds to the designed collaborative strategy. Based on a qualitative evaluation, we found that volunteers were satisfied with the procedure used to create AD and visually impaired users felt that this AD assisted them.

Saliency-weighted graphs for efficient visual content description and their applications in real-time image retrieval systems

The exponential growth in the volume of digital image databases is making it increasingly difficult to retrieve relevant information from them. Efficient retrieval systems require distinctive features extracted from visually rich contents, represented semantically in a human perception-oriented manner. This paper presents an efficient framework to model image contents as an undirected attributed relational graph, exploiting color, texture, layout, and saliency information. The proposed method encodes salient features into this rich representative model without requiring any segmentation or clustering procedures, reducing the computational complexity. In addition, an efficient graph-matching procedure implemented on specialized hardware makes it more suitable for real-time retrieval applications. The proposed framework has been tested on three publicly available datasets, and the results prove its superiority in terms of both effectiveness and efficiency in comparison with other state-of-the-art schemes.

Image region description using orthogonal combination of local binary patterns enhanced with color information

Visual content description is a key issue for machine-based image analysis and understanding. A good visual descriptor should be both discriminative and computationally efficient while possessing some properties of robustness to viewpoint changes and lighting condition variations. In this paper, we propose a new operator called the orthogonal combination of local binary patterns (denoted as OC-LBP) and six new local descriptors based on OC-LBP enhanced with color information for image region description. The aim is to increase both discriminative power and photometric invariance properties of the original LBP operator while keeping its computational efficiency. The experiments in three different applications show that the proposed descriptors outperform the popular SIFT, CS-LBP, HOG and SURF, and achieve comparable or even better performances than the state-of-the-art color SIFT descriptors. Meanwhile, the proposed descriptors provide complementary information to color SIFT, because a fusion of these two kinds of descriptors is found to perform clearly better than either of the two separately. Moreover, the proposed descriptors are about four times faster to compute than color SIFT.

A new model for semantic photograph description combining basic levels and user-assigned descriptors

Few studies have been conducted to identify users’ desired semantic levels of image access when describing, searching, and retrieving photographs online. The basic level, or the level of abstraction most commonly used to describe an item, is a cognitive theory currently under consideration in image retrieval research. This study investigates potential basic levels of description for online photographs by testing the Hierarchy for Online Photograph Representation (HOPR) model, which is based on a need for a model that addresses users’ basic levels of photograph description and retrieval. We developed the HOPR model using the following three elements as guides: the most popular tags of all time on Flickr, the Pyramid model for visual content description by Jörgensen, Jaimes, Benitez, and Chang, and the nine classes of image content put forth by Burford, Briggs, and Eakins. In an exploratory test of the HOPR model, participants were asked to describe their first reaction to, and possible free-text indexing terms for, a small set of personal photographs. Content analysis of the data indicated a clear set of user preferences that are consistent with prior image description studies. Generally speaking, objects in the photograph and events taking place in the photograph were the most commonly used levels of description. The preliminary HOPR model shows promise for its intended utility, but further refinement is needed through additional research.

Real-time visual content description system based on MPEG-7 descriptors

This paper presents a real-time Visual Content Description System (VCDS) based on MPEG-7 descriptors. In our approach the system's structure is divided into two parts, the first of which is the extraction of the descriptors using the VCDS. The second part uses the descriptors' values in a particular search algorithm. We propose here original solutions for both parts. The proposed system architecture could be used for real-time video indexing and retrieval, content summarization, content delivery, surveillance, personalized services, etc. The descriptor extractor IP core, which is part of the VCDS, implements four MPEG-7 visual descriptors and was designed for ASIC implementation in CMOS 0.35 μm, which is a novel solution for a real-time content description problem. The proposed hardware architecture splits the computational burden into several threads, so that calculations are made simultaneously in order to improve the system's speed. These methods make the hardware implementation of the most computationally demanding modules of the system more time- and power-efficient. Four different variations of the basic hardware architecture are discussed. New search algorithms based on the VCDS responses are also proposed. Experimental results demonstrate the effectiveness of the hardware architectures, and the new approach to similarity-based searching methods.

Adaptive bitstream switching of scalable video

With scalable video coding that provides fine-granular quality degradation, such as fine granularity scalability (FGS) and progressive FGS (PFGS), or H.264 scalable video coding's (SVC) adaptive reference FGS (AR-FGS) coding, video can flexibly be streamed to receivers of heterogeneous bandwidths. However, the transmitted video is only efficiently encoded when the transmission bit rate is in the vicinity of the encoding bit rate. In this paper, we develop and evaluate a comprehensive suite of network-aware adaptive bitstream switching policies for point-to-point and point-to-multipoint streaming of fine granular scalable coded video to address this coding efficiency issue. Our approach stores a small number of encodings (versions) with different encoding bit rates for each video sequence and estimates the reconstructed quality using the motion activity levels of the underlying visual content (or, in general, any content descriptor(s) that highly correlate with the reconstructed quality). For unicast streaming, we then: (i) adaptively switch between the different encodings at the server, to improve the reconstructed video quality and (ii) adaptively drop packets during network congestion to ensure fairness between multiple unicast streams. For multicast streaming, we also adaptively switch between the different encodings to maximize the average video quality. Our adaptive bitstream switching policies consider the visual content descriptors as well as the network channel variability, while requiring only sample points from the rate-distortion curve of the video stream. From our extensive simulations with PFGS coding, we find that our adaptive unicast bitstream switching policy achieves on average a 0.8 dB improvement over the optimal non-adaptive streaming for a diverse 200-shot sequence from Star Wars IV. We have also verified our key findings with the latest scalable video coding standard, H.264 SVC.

Semantic-associative visual content labelling and retrieval: A multimodal approach

A novel framework referred to as collaterally confirmed labelling (CCL) is proposed, aiming at localising the visual semantics to regions of interest in images with textual keywords. Both the primary image and collateral textual modalities are exploited in a mutually co-referencing and complementary fashion. The collateral content and context-based knowledge is used to bias the mapping from the low-level region-based visual primitives to the high-level visual concepts defined in a visual vocabulary. We introduce the notion of collateral context, which is represented as a co-occurrence matrix of the visual keywords. A collaborative mapping scheme is devised using statistical methods like Gaussian distribution or Euclidean distance together with collateral content and context-driven inference mechanism. We introduce a novel high-level visual content descriptor that is devised for performing semantic-based image classification and retrieval. The proposed image feature vector model is fundamentally underpinned by the CCL framework. Two different high-level image feature vector models are developed based on the CCL labelling of results for the purposes of image data clustering and retrieval, respectively. A subset of the Corel image collection has been used for evaluating our proposed method. The experimental results to-date already indicate that the proposed semantic-based visual content descriptors outperform both traditional visual and textual image feature models.

Chomsky and Egan on computational theories of vision

Noam Chomsky and Frances Egan argue that David Marr's computational theory of vision is not intentional, claiming that the formal scientific theory does not include description of visual content. They also argue that the theory is internalist in the sense of not describing things physically external to the perceiver. They argue that these claims hold for computational theories of vision in general. Beyond theories of vision, they argue that representational content does not figure as a topic within formal computational theories in cognitive science. I demonstrate that Chomsky's and Egan's claims about Marr's theory are false. Marr's computational theory contains a mathematical theory of visual content, based on empirical psychophysical evidence. It also contains mathematical descriptions of distal physical surfaces and objects, and of their optic projection to the perceiver. Much computational research on vision contains these types of intentional and externalist components within the formal, mathematical, theories. Chomsky's and Egan's claims demonstrate inadequate study and understanding of Marr's work and other research in this area. Computational theories of vision, by containing empirically based mathematical theories of visual content, to this extent present naturalizations of semantics.

A Framework for Advanced Video Traces: Evaluating Visual Quality for Video Transmission Over Lossy Networks

Conventional video traces (which characterize the video encoding frame sizes in bits and frame quality in PSNR) are limited to evaluating loss-free video transmission. To evaluate robust video transmission schemes for lossy network transport, generally experiments with actual video are required. To circumvent the need for experiments with actual videos, we propose in this paper an advanced video trace framework. The two main components of this framework are (i) advanced video traces which combine the conventional video traces with a parsimonious set of visual content descriptors, and (ii) quality prediction schemes that based on the visual content descriptors provide an accurate prediction of the quality of the reconstructed video after lossy network transport. We conduct extensive evaluations using a perceptual video quality metric as well as the PSNR in which we compare the visual quality predicted based on the advanced video traces with the visual quality determined from experiments with actual video. We find that the advanced video trace methodology accurately predicts the quality of the reconstructed video after frame losses.

A Foundational Perspective for Visual Information Retrieval

The author discusses a new book on visual information retrieval (VIR). The fundamental components include visual content description and representation, similarity and distance measures, classifiers, indexing schemes, user interaction, and system performance evaluation.

EFFICIENT CONTENT-BASED IMAGE RETRIEVAL USING FUZZY ORGANIZATION AND OPTIMAL RELEVANCE FEEDBACK

The performance of a Content-Based Image Retrieval System (CBIR) depends on (a) the system's adaptability to the user's information needs, which permits different types of indexing and simultaneously reduces the subjectivity of human perception for the interpretation of the image visual content and (b) the efficient organization of the extracted descriptors, which represent the rich visual information. Both issues are addressed in this paper. Descriptor organization is performed using a fuzzy classification scheme fragmented into multidimensional classes, instead of the previous works where fuzzy histograms were created in one dimension using, for example, the feature vector norm. Multidimensionality relates the descriptors with one another and thus allows a compact and meaningful visual representation by mapping the elements of the resulted feature vectors with a physical visual interpretation. Furthermore, fuzzy classification is applied for all visual content descriptors, in contrast to the previous approaches where only color information is exploited. Two kinds of content descriptors are extracted in our case; global-based and region-based. The first refers to the global image characteristics, while the second exploits the region-based properties. Regions are obtained by applying a multiresolution implementation of the Recursive Shortest Spanning Tree (RSST) algorithm, called M-RSST in this paper. The second issue is addressed by proposing a computationally efficient relevance feedback mechanism based on an optimal weight updating strategy. The scheme relies on the cross-correlation measure, instead of the Euclidean distance which is mainly used in most relevance feedback algorithms. Cross-correlation is a normalized measure, which expresses how similar the two feature vectors are and thus it indicates a metric of their content similarity. The proposed scheme can be recursively implemented in the case of multiple feedback iterations, instead of the previous approaches. Furthermore, it provides reliable results regardless of the number of selected sample and the feature vector size improving relevance feedback performance, as compared to other approaches.

PicSOM-self-organizing image retrieval with MPEG-7 content descriptors

Development of content-based image retrieval (CBIR) techniques has suffered from the lack of standardized ways for describing visual image content. Luckily, the MPEG-7 international standard is now emerging as both a general framework for content description and a collection of specific agreed-upon content descriptors. We have developed a neural, self-organizing technique for CBIR. Our system is named PicSOM and it is based on pictorial examples and relevance feedback (RF). The name stems from "picture" and the self-organizing map (SOM). The PicSOM system is implemented by using tree structured SOMs. In this paper, we apply the visual content descriptors provided by MPEG-7 in the PicSOM system and compare our own image indexing technique with a reference system based on vector quantization (VQ). The results of our experiments show that the MPEG-7-defined content descriptors can be used as such in the PicSOM system even though Euclidean distance calculation, inherently used in the PicSOM system, is not optimal for all of them. Also, the results indicate that the PicSOM technique is a bit slower than the reference system in starting to find relevant images. However, when the strong RF mechanism of PicSOM begins to function, its retrieval precision exceeds that of the reference system.

The MPEG-7 visual standard for content description-an overview

The MPEG-7 visual standard under development specifies content-based descriptors that allow users or agents (or search engines) to measure similarity in images or video based on visual criteria, and can be used to efficiently identify, filter, or browse images or video based on visual content. More specifically, MPEG-7 specifies color, texture, object shape, global motion, or object motion features for this purpose. This paper outlines the aim, methodologies, and broad details of the MPEG-7 standard development for visual content description.

A conceptual framework and empirical research for classifying visual descriptors

AbstractThis article presents exploratory research evaluating a conceptual structure for the description of visual content of images. The structure, which was developed from empirical research in several fields (e.g., Computer Science, Psychology, Information Studies, etc.), classifies visual attributes into a “Pyramid” containing four syntactic levels (type/technique, global distribution, local structure, composition), and six semantic levels (generic, specific, and abstract levels of both object and scene, respectively). Various experiments are presented, which address the Pyramid's ability to achieve several tasks: (1) classification of terms describing image attributes generated in a formal and an informal description task, (2) classification of terms that result from a structured approach to indexing, and (3) guidance in the indexing process. Several descriptions, generated by naive users and indexers, are used in experiments that include two image collections: a random Web sample, and a set of news images. To test descriptions generated in a structured setting, an Image Indexing Template (developed independently over several years of this project by one of the authors) was also used. The experiments performed suggest that the Pyramid is conceptually robust (i.e., can accommodate a full range of attributes), and that it can be used to organize visual content for retrieval, to guide the indexing process, and to classify descriptions obtained manually and automatically.

A conceptual framework and empirical research for classifying visual descriptors

This article presents exploratory research evaluating a conceptual structure for the description of visual content of images. The structure, which was developed from empirical research in several fields (e.g., Computer Science, Psychology, Information Studies, etc.), classifies visual attributes into a Pyramid containing four syntactic levels (type/technique, global distribution, local structure, composition), and six semantic levels (generic, specific, and abstract levels of both object and scene, respectively). Various experiments are presented, which address the Pyramid's ability to achieve several tasks: (1) classification of terms describing image attributes generated in a formal and an informal description task, (2) classification of terms that result from a structured approach to indexing, and (3) guidance in the indexing process. Several descriptions, generated by naive users and indexers, are used in experiments that include two image collections: a random Web sample, and a set of news images. To test descriptions generated in a structured setting, an Image Indexing Template (developed independently over several years of this project by one of the authors) was also used. The experiments performed suggest that the Pyramid is conceptually robust (i.e., can accommodate a full range of attributes), and that it can be used to organize visual content for retrieval, to guide the indexing process, and to classify descriptions obtained manually and automatically.

Generalized predictive binary shape coding using polygon approximation

Abstract This paper introduces a new binary shape coding technique called generalized predictive shape coding (GPSC) to encode the boundary of a visual object compactly by using a vertex-based approach. GPSC consists of a contour pixel matching algorithm and a motion-compliant contour coding algorithm. The contour pixel matching algorithm utilizes the knowledge of previously decoded contours by using a uniform translational model for silhouette motion, and generalizes polygon approximation for lossless and lossy motion estimation by adjusting a tolerance parameter d max . To represent motion-compliant regions with minimum information in the transmitted bitstream, we develop a reference index-based coding scheme to represent the 2D positions of the matched segments using 1D reference contour indices. Finally, we encode the mismatched segments by sending residual polygons until the distortion is less than d max . While GPSC realizes polygon approximation exactly at every encoding stage, we can guarantee quality of service because the peak distortion is no greater than d max , and we improve coding efficiency as long as a silhouette complies with the model. The tolerance parameter d max can be assigned to each contour to smooth the transmitted data rate, which is especially useful for constant bandwidth channels. Compared with non-predictive approaches, simulation using MPEG-4 sequences demonstrates that GPSC not only improves objective gain but also enhances visual quality based on MPEG-4 subjective tests. The significance of GPSC is that it provides a generic framework for seamlessly extending conventional vertex coding schemes into the temporal domain yet it retains the advantages of existing polygon-based algorithms for visual content description while furnishing better geometric compression.

Indexed retrieval by shape appearance

Efficient retrieval by content of visual information requires that visual content descriptors and similarity models are combined with efficient index structures. This problem is particularly challenging in the case of retrieval by shape similarity. The paper discusses retrieval by shape similarity, using local features and effective indexing. Shapes are partitioned into tokens following curvature analysis and each token is modelled by a set of perceptually salient attributes. Two distinct distance functions are used to model token similarity and shape similarity. Shape indexing is obtained by arranging tokens into an M-tree index structure. Examples from a prototype system and computational experiences are reported for both retrieval accuracy and indexing efficiency.