Multimedia Objects Research Articles

A parallel disk storage system for real-time multimedia applications

We describe design and implementation of the RIO (randomized I/O) multimedia object server which manages a set of parallel disks and supports real-time throughput and statistical delay guarantees. This storage subsystem was implemented as part of a multimedia information server supporting multiple concurrent applications such as video on demand and 3D interactivc virtual worlds. We discuss principal issues and innovations involved in design and implementation of RIO storage system, presenting experimental performance results measured on a prototype implementation. A multimedia data server must be ready to handle a variety of real-time object types (video, audio, 3D interactive virtual worlds, etc.) along with a nonreal-time workload component. Achieving simultaneously (1) high utilization and (2) low latency with high degree of certainty is the challenge. Our prototype system provides a statistical guarantee of quality of service. Our experimental results show that it is possible to achieve a very small probability of exceeding a deadline (less than 10 -6 ), with relatively high disk utilization (70-99%, in terms of fraction of the maximum disk capacity), together with relatively small delay bound (on the order of 0.5-1.5 s), using contemporary disks. Results were achieved using random allocation of disk blocks and replication in conjunction with on-line load balancing.

Querying multimedia presentations

We examine the querying requirements of large libraries of multimedia presentations. Given these requirements, we examine work on querying temporal and sequence data, and identify the similarities and dissimilarities between presentations and these other forms of data. From this analysis, we propose an integrated composition and query capability to permit the reuse of multimedia objects, presentations and presentation segments. The query facility permits content and attribute based queries along with queries over temporal synchronisation characteristics. The main contributions of our work are: it addresses both determinant and indeterminant intervals; it permits querying over presentation libraries with heterogeneous structure; it builds on work from temporal and sequence database to address the unique semantics of presentations.

Spatio-temporal composition and indexing for large multimedia applications

Multimedia applications usually involve a large number of multimedia objects (texts, images, sounds, etc.). An important issue in this context is the specification of spatial and temporal relationships among these objects. In this paper we define such a model, based on a set of spatial and temporal relationships between objects participating in multimedia applications. Our work exploits existing approaches for spatial and temporal relationships. We extend these relationships in order to cover the specific requirements of multimedia applications and we integrate the results in a uniform framework for spatio-temporal composition representation. Another issue is the efficient handling of queries related to the spatio-temporal relationships among the objects during the authoring process. Such queries may be very costly and appropriate indexing schemes are needed so as to handle them efficiently. We propose efficient such schemes, based on multidimensional (spatial) data structures, for large multimedia applications that involve thousands of objects. Evaluation models of the proposed schemes are also presented, as well as hints for the selection of the most appropriate one, according to the multimedia author's requirements.

Retrieval schedules based on resource availability and flexible presentation specifications

A distributed multimedia document presentation involves retrieval of objects from one or more document servers and their presentation at the client system. The presentation of the multimedia objects has to be carried out in accordance with the specification of temporal relationships between the objects. The retrieval of multimedia objects from the document server(s) is influenced by factors such as temporal specification of objects presentations, throughput offered by the network service provider, and the buffer resources on the client system. Flexibility in the temporal specification of the multimedia document may help in deriving an object retrieval schedule that can handle variations in network throughput and buffer resource availability. In this paper, we develop techniques for deriving a flexible object retrieval schedule for a distributed multimedia document presentation. The schedule is based on flexible temporal specification of the multimedia document using the difference constrai nts approach. We show how the derived retrieval schedule can be validated and modified to ensure that it can work with the offered network throughput and the available buffer resources.

Dimensionality reduction for similarity searching in dynamic databases

Databases are increasingly being used to store multi-media objects such as maps, images, audio and video. Storage and retrieval of these objects is accomplished using multi-dimensional index structures such as R*-trees and SS-trees. As dimensionality increases, query performance in these index structures degrades. This phenomenon, generally referred to as the dimensionality curse, can be circumvented by reducing the dimensionality of the data. Such a reduction is however accompanied by a loss of precision of query results. Current techniques such as QBIC use SVD transform-based dimensionality reduction to ensure high query precision. The drawback of this approach is that SVD is expensive to compute, and therefore not readily applicable to dynamic databases. In this paper, we propose novel techniques for performing SVD-based dimensionality reduction in dynamic databases. When the data distribution changes considerably so as to degrade query precision, we recompute the SVD transform and incorporate it in the existing index structure. For recomputing the SVD-transform, we propose a novel technique that uses aggregate data from the existing index rather than the entire data. This technique reduces the SVD-computation time without compromising query precision. We then explore efficient ways to incorporate the recomputed SVD-transform in the existing index structure without degrading subsequent query response times. These techniques reduce the computation time by a factor of 20 in experiments on color and texture image vectors. The error due to approximate computation of SVD is less than 10%.

Synchronization of distributed multimedia systems with user interactions

Without user interactions, multimedia presentations are just fancy slide shows with sound and video supports. User interactions by themselves do not change the temporal relationships among multimedia objects, such as texts, graphics, images, audio, and video, but affect the playback schedules. In this paper, we propose a synchronization mechanism to guarantee the quality of multimedia presentation with user interactions. In our protocol, each presentation site requests media transmission from the required media servers at certain time intervals prior to the playback deadlines, where these time intervals are the response times to cover possible experienced end-to-end delays and packet losses, and waits for an initial setup time to ensure intermedia synchronization before starting the presentation. Users may interact with the presentation. This synchronization mechanism solves the problems incurred by user interactions by determining the new presentation scenario produced by the interactive operation, calculating the corresponding setup time, and then rendering the new playback and retrieval schedules.

Let's talk about it: dialogues with multimedia databases: Database support for human activity

We describe two scenarios of user tasks in which access to multimedia data plays a significant role. Because current multimedia databases cannot support these tasks, we introduce three new requirements on multimedia databases: multimedia objects should be active objects, querying is an interaction process, and query processing uses multiple representations. We discuss three techniques to handle multimedia objects as active objects. Also, we introduce a promising database architecture to meet the new user requirements. Agents within the database handle objects' representations, and a search engine on top of a conventional database handles relevance feedback and multiple representations.

Modelling and Verification of PREMO Synchronisable Objects

Abstract. The PREMO standard, Presentation Environment for Multimedia Objects, is a major new standard under development within ISO/IEC. It addresses the creation of, presentation of, and interaction with all forms of information using single or multiple media. In this paper we give a formal LOTOS specification, amenable to automatic verification, of the PREMO synchronisable object, which is one of the central parts of the standard. Various design options are investigated by a combination of constraint oriented specification and model checking. This shows the usefulness of formal specification and automatic verification during the design phase of an international standard.

Multimedia—An introduction

Multimedia--the combination of text, animated graphics, video, and sound--presents information in a way that is more interesting and easier to grasp than text alone. It has been used for education at all levels, job training, and games and by the entertainment industry. It is becoming more readily available as the price of personal computers and their accessories declines. Multimedia as a human-computer interface was made possible some half-dozen years ago by the rise of affordable digital technology. Previously, multimedia effects were produced by computer-controlled analog devices, like videocassette recorders, projectors, and tape recorders. Digital technology's exponential decline in price and increase in capacity has enabled it to overtake analog technology. The Internet is the breeding ground for multimedia ideas and the delivery vehicle of multimedia objects to a huge audience. This paper reviews the uses of multimedia, the technologies that support it, and the larger architectural and design issues.

Conceptual Modeling and Querying inMultimedia Databases

In this paper, we discuss some design principles that will aid in the design and querying of multimedia databases. We use an object-relational data model and argue that multimedia objects should normally have a special attribute called ‘core’ which stores the real object itself in contrast to the abstraction which is reflected in the rest of the attributes. We present an extension to the ER Diagram that takes advantage of the ‘core’ notion to facilitate design of multimedia databases. We discuss some desirable features in a query language for multimedia databases: simplifications like the use of path expressions and implicit use of functions (methods) as attributes, and explicit specification of the display layout and format either at the data definition level or query specification level. To materialize this last feature, we propose a display specification extension to SQL (SQL+D) that we have implemented.

Supporting ranked Boolean similarity queries in MARS

To address the emerging needs of applications that require access to and retrieval of multimedia objects, we are developing the Multimedia Analysis and Retrieval System (MARS). In this paper, we concentrate on the retrieval subsystem of MARS and its support for content-based queries over image databases. Content-based retrieval techniques have been extensively studied for textual documents in the area of automatic information retrieval. This paper describes how these techniques can be adapted for ranked retrieval over image databases. Specifically, we discuss the ranking and retrieval algorithms developed in MARS based on the Boolean retrieval model and describe the results of our experiments that demonstrate the effectiveness of the developed model for image retrieval.

Content-based retrieval in digital libraries

With the recent developments in multimedia and telecommunication technologies, content-based information is becoming increasingly important for various areas such as digital libraries, interactive video and multimedia publishing. Multimedia data refers to simple structured data (such as numbers and short strings), large unstructured data (such as text documents, images, audio and video data) and complex structured data (such as maps, graphs, charts and tables). In this article, we briefly address content-based retrieval and the issues of representation, storage and retrieval of multimedia objects in digital libraries. We then very briefly identify some open areas of research.

Relating the primitive hierarchy of the PREMO standard to the standard reference model for intelligent multimedia presentation systems

The need for a suitable classification of media types arises for several reasons when building or comparing multimedia systems. Within an Intelligent Multimedia Presentation Systems (IMMPS), it is necessary to formulate and encode design knowledge for decision making on the appropriate medium in which to present information and for the generation of the presentation. It is also required in order to specify interfaces to and between system components which will be employed to run a generated presentation before the user's eyes. This task is reflected in the Standard Reference Model (SRM, see this volume) for IMMPS by the Presentation Display Layer. However, the SRM does not instantiate this layer in detail, but instead refers to the Presentation Environment for Multimedia Objects (PREMO) ISO/IEC standard which provides a reference model for a presentation runtime environment for multimedia. PREMO already contains a set of basic structures, the so-called PREMO Primitive Hierarchy, to describe different media types. Thus the question arises, as to how far the PREMO Primitive Hierarchy could serve as a media classification for the SRM in general. In particular, this would support consistency between the design and presentation layers of the SRM if PREMO were used to instantiate the presentation layer. In the current paper, we first point to a number of typical problems with generating classifications of media types. We then provide a brief introduction to PREMO and its Primitive Hierarchy. Finally, the benefits and costs of using the PREMO Primitive Hierarchy for the SRM are discussed.

Tertiary storage in multimedia systems: staging or direct access?

Multimedia applications that are required to manipulate large collections of objects are becoming increasingly common. Moreover, the size of multimedia objects, which are already huge, are getting even bigger as the resolution of output devices improve. As a result, many multimedia storage systems are not likely to be able to keep all of their objects disk-resident. Instead, a majority of the less popular objects have to be off-loaded to tertiary storage to keep costs down. The speed at which objects can be accessed from tertiary storage is thus an important consideration. In this paper, we propose an adaptive data retrieval algorithm that employs a combination of staging and direct access in servicing tertiary storage retrieval requests. At retrieval time, an object that resides in tertiary storage can either be staged to and then played back from disks, or the object can be accessed directly from the tertiary drives. We show that a simplistic policy that adheres strictly to staging or direct access does not exploit the full retrieval capacity of both the tertiary library and the secondary storage. To overcome the problem, we propose a data retrieval algorithm that dynamically chooses between staging and direct access, based on the relative load on the tertiary versus secondary devices. A series of simulation experiments confirms that the algorithm achieves good access times over a wide range of workloads and resource configurations. Moreover, the algorithm is very responsive to changing load conditions.

The Open Inventor Toolkit and the PREMO Standard

PREMO is an emerging international standard for the presentations of multimedia objects including computer graphics. Open Inventor™ is a commercially available “de facto” standard for interactive computer graphics packaged as a library of objects. In this paper, we consider whether the concepts and objects of PREMO are sufficient to represent a professional quality system, such as Open Inventor.By comparing PREMO with Open Inventor, we hope to show that PREMO's computer graphics environment model and event model can properly describe Open Inventor's rendering action and event model. The scene graph is very important in Open Inventor. Most Open Inventor functions rely on various operations over scene graphs. The construction, edition and traversal of the scene graphs are implemented as a set of newly defined PREMO objects. Graphics rendering, event handling and scene graphs constitute the fundamental parts of Open Inventor, other Open Inventor functionalities can be constructed from these. We conclude that since these three fundamental parts of Open Inventor can be properly modelled and implemented by means of PREMO, that the concepts and objects of PREMO are sufficient to represent Open Inventor.

RIO

A multimedia object server must be ready to handle a variety of media object types (video, audio, image, 3D interactive, etc.) as well as non real-time workload. Even when a homogeneous set of object types are maintained in the store (e.g., all videos) the storage system workload is generally quite variable due to the need to provide for example, VCR functionality, multiple playout rates, different resolution levels for the same objects, etc. Attempting to carefully layout data and optimally schedule delivery to meet just-in-time delivery constraints is very difficult in the face of this heterogeneous workload. Our approach to the unpredictability of the I/O workload is to randomize the allocation of disk blocks. This turns all workloads into the same uniformly random access pattern and thus gives one problem to deal with. The main disadvantage of this approach is that statistical variation can result in short term imbalances in disk utilization which in turn, cause large variances in latencies. Our approach to this problem is to introduce limited redundancy and asynchronous scheduling for short term load balancing.This approach is being implemented in the RIO (Random I/O) multimedia object server. The RIO multimedia object server provides applications a guaranteed rate of storage access with bounded delay even at very high ( > 90% ) disk utilization.

Scheduling and data layout policies for a near-line multimedia storage architecture

Recent advances in computer technologies have made it feasible to provide multimedia services, such as news distribution and entertainment, via high-bandwidth networks. The storage and retrieval of large multimedia objects (e.g., video) becomes a major design issue of the multimedia information system. While most other works on multimedia storage servers assume an on-line disk storage system, we consider a two-tier storage architecture with a robotic tape library as the vast near-line storage and an on-line disk system as the front-line storage. Magnetic tapes are cheaper, more robust, and have a larger capacity; hence, they are more cost effective for large scale storage systems (e.g., videoon-demand (VOD) systems may store tens of thousands of videos). We study in detail the design issues of the tape sub-system and propose some novel tape-scheduling algorithms which give faster response and require less disk buffer space. We also study the disk-striping policy and the data layout on the tape cartridge in order to fully utilize the throughput of the robotic tape system and to minimize the on-line disk storage space.

Distributed multimedia synchronization specifications using M 2EST

In order to properly schedule related multimedia objects, synchronization relationships of multimedia objects should be precisely specified and dispatched. Each multimedia presentation schedule contains two parts: (1) the state-transition control part, which specifies intra-medium and inter-media synchronization information, and (2) the data variables part, which specifies the dynamic aspects of the state-transition control for dealing with jitter and skew. In this paper, we propose a specification language for specifying multimedia synchronization. The language is called M 2EST, which represents the MultiMedia Extended State Transition. M 2EST can handle both the state-transition control part and the data variables part in multimedia presentation scheduling. Using M 2EST, the temporal behavior of each medium stream is handled by an actor extended finite state machine (EFSM). The temporal relationships among media streams are handled by a synchronizer EFSM. Synchronizer and actors perform multimedia presentations cooperatively. The corresponding synchronization schemes, including both intra-stream and inter-stream synchronization schemes, which rectify the random networks delays caused on distributed presentation environment, can also be specified using M 2EST.

From multimedia models to multimedia transport protocols

This paper presents a new protocol architecture for distributed multimedia systems which includes a new and general protocol layer for the transport of multimedia objects. The proposed protocols are based on the use of two extended timed and time Petri net models that specify the synchronisation requirements of multimedia applications. After presenting these two formal models, it is shown how they have been used to design the general multimedia architecture whose transport layer is based on the concept of a partial order connection, an extension of the classical connection oriented (e.g., TCP) and connectionless (e.g., UDP) transmission concepts.

DEVise (demo abstract)

DEVise is a data exploration system that allows users to easily develop, browse, and share visual presentations of large tabular datasets (possibly containing or referencing multimedia objects) from several sources. The DEVise framework, implemented in a tool that has been already successfully applied to a variety of real applications by a number of user groups, makes several contributions. In particular, it combines support for extended relational queries with powerful data visualization features. Datasets much larger than available main memory can be handled—DEVise is currently being used to visualize datasets well in excess of 100MB—and data can be interactively examined at several levels of detail: all the way from meta-data summarizing the entire dataset, to large subsets of the actual data, to individual data records. Combining querying (in general, data processing) with visualizations gives us a very versatile tool, and presents several novel challenges. Our emphasis is on developing an intuitive yet powerful set of querying and visualization primitives that can be easily combined to develop a rich set of visual presentations that integrate data from a wide range of application domains. In this demo, we will present a number of examples of the use of the DEVise tool for visualizing and interactively exploring very large datasets, and report on our experience in applying it to several real applications.