Image-based Representation Research Articles

The Design and Construction of a Movable Image-Based Rendering System and Its Application to Multiview Conferencing

Image-based rendering (IBR) is an promising technology for rendering photo-realistic views of scenes from a collection of densely sampled images or videos. It provides a framework for developing revolutionary virtual reality and immersive viewing systems. While there has been considerable progress recently in the capturing, storage and transmission of image-based representations, most multiple camera systems are designed to be stationary and hence their ability to cope with moving objects and dynamic environment is somewhat limited. This paper studies the design and construction of a movable image-based rendering system based on a class of dynamic representations called plenoptic videos, its associated video processing algorithms and an application to multiview audio-visual conferencing. It is constructed by mounting a linear array of 8 video cameras on an electrically controllable wheel chair and its motion is controllable manually or remotely through wireless LAN by means of additional hardware circuitry. We also developed a real-time object tracking algorithm and utilize the motion information computed to adjust continuously the azimuth or rotation angle of the movable IBR system in order to cope with a given moving object in a large environment. Due to imperfection in tracking and mechanical vibration encountered in movable systems, the videos may appear very shaky and a new video stabilization technique is proposed to overcome this problem. The usefulness of the system is illustrated by means of a multiview conferencing application using a multiview TV display. Through this pilot study, we hope to disseminate useful experience for the design and construction of movable IBR systems with improved viewing freedom and ability to cope with moving object in a large environment.

Object-Based Coding for Plenoptic Videos

A new object-based coding system for a class of dynamic image-based representations called plenoptic videos (PVs) is proposed. PVs are simplified dynamic light fields, where the videos are taken at regularly spaced locations along line segments instead of a 2-D plane. In the proposed object-based approach, objects at different depth values are segmented to improve the rendering quality. By encoding PVs at the object level, desirable functionalities such as scalability of contents, error resilience, and interactivity with an individual image-based rendering (IBR) object can be achieved. Besides supporting the coding of texture and binary shape maps for IBR objects with arbitrary shapes, the proposed system also supports the coding of grayscale alpha maps as well as depth maps (geometry information) to respectively facilitate the matting and rendering of the IBR objects. Both temporal and spatial redundancies among the streams in the PV are exploited to improve the coding performance, while avoiding excessive complexity in selective decoding of PVs to support fast rendering speed. Advanced spatial/temporal prediction methods such as global disparity-compensated prediction, as well as direct prediction and its extensions are developed. The bit allocation and rate control scheme employing a new convex optimization-based approach are also introduced. Experimental results show that considerable improvements in coding performance are obtained for both synthetic and real scenes, while supporting the stated object-based functionalities.

A Process Model for Direct Correlation between Computed Tomography and Histopathology: Application in Lung Cancer

Multimodal imaging techniques for capturing normal and diseased human anatomy and physiology are being developed to benefit patient clinical care, research, and education. In the past, the incorporation of histopathology into these multimodal datasets has been complicated by the large differences in image quality, content, and spatial association. We have developed a novel system, the large-scale image microtome array (LIMA), to bridge the gap between nonstructurally destructive and destructive imaging such that reliable registration between radiological data and histopathology can be achieved. Registration algorithms have been designed to align the multimodal datasets, which include computed tomography, computed micro-tomography, LIMA, and histopathology data to a common coordinate system. The resulting volumetric dataset provides an abundance of valuable information relating to the tissue sample including density, anatomical structure, color, texture, and cellular information in three dimensions. An image processing pipeline has been established to register all the multimodal data to a common coordinate system. In this study, we have chosen to use human lung cancer nodules as an example; however, the flexibility of the image acquisition and subsequent processing algorithms makes it applicable to any soft organ tissue. A novel process model has been established to generate cross registered multimodal datasets for the investigation of human lung cancer nodule content and associated image-based representation.

Touchable 3D video system

Multimedia technologies are reaching the limits of providing audio-visual media that viewers consume passively. An important factor, which will ultimately enhance the user's experience in terms of impressiveness and immersion, is interaction. Among daily life interactions, haptic interaction plays a prominent role in enhancing the quality of experience of users, and in promoting physical and emotional development. Therefore, a critical step in multimedia research is expected to bring the sense of touch, or haptics, into multimedia systems and applications. This article proposes a touchable 3D video system where viewers can actively touch a video scene through a force-feedback device, and presents the underlying technologies in three functional components: (1) contents generation, (2) contents transmission, and (3) viewing and interaction. First of all, we introduce a depth image-based haptic representation (DIBHR) method that adds haptic and heightmap images, in addition to the traditional depth image-based representation (DIBR), to encode the haptic surface properties of the video media. In this representation, the haptic image contains the stiffness, static friction, and dynamic friction, whereas the heightmap image contains roughness of the video contents. Based on this representation method, we discuss how to generate synthetic and natural (real) video media through a 3D modeling tool and a depth camera, respectively. Next, we introduce a transmission mechanism based on the MPEG-4 framework where new MPEG-4 BIFS nodes are designed to describe the haptic scene. Finally, a haptic rendering algorithm to compute the interaction force between the scene and the viewer is described. As a result, the performance of the haptic rendering algorithm is evaluated in terms of computational time and smooth contact force. It operates marginally within a 1 kHz update rate that is required to provide stable interaction force and provide smoother contact force with the depth image that has high frequency geometrical noise using a median filter.

An object-based approach to image/video-based synthesis and processing for 3-D and multiview televisions

This paper proposes an object-based approach to a class of dynamic image-based representations called videos, where the plenoptic video sequences are segmented into image-based renderin...

An Object-Based Approach to Image/Video-Based Synthesis and Processing for 3-D and Multiview Televisions

This paper proposes an object-based approach to a class of dynamic image-based representations called ldquoplenoptic videos,rdquo where the plenoptic video sequences are segmented into image-based rendering (IBR) objects each with its image sequence, depth map, and other relevant information such as shape and alpha information. This allows desirable functionalities such as scalability of contents, error resilience, and interactivity with individual IBR objects to be supported. Moreover, the rendering quality in scenes with large depth variations can also be improved considerably. A portable capturing system consisting of two linear camera arrays was developed to verify the proposed approach. An important step in the object-based approach is to segment the objects in video streams into layers or IBR objects. To reduce the time for segmenting plenoptic videos under the semiautomatic technique, a new object tracking method based on the level-set method is proposed. Due to possible segmentation errors around object boundaries, natural matting with Bayesian approach is also incorporated into our system. Furthermore, extensions of conventional image processing algorithms to these IBR objects are studied and illustrated with examples. Experimental results are given to illustrate the efficiency of the tracking, matting, rendering, and processing algorithms under the proposed object-based framework.

Using relief texture for interactive and tangible virtual environments

This paper first introduces a way to improve interactivity with high polygon count virtual objects through the “mixed” use of image-based representation within one object. That is, both 3D polygonal and image-based representations are maintained for an object, and switched for rendering depending on the functional requirement of the object. Furthermore, in order to reduce the popping effect and provide smooth and gradual transition during the object representation switch, the object is subdivided with the subdivided parts possibly represented differently, i.e., using 3D models or images. As for the image-based representation, the relief texture (RT) method is used. In particular, through the use of the mixed representation, a new way called TangibleScreen is proposed to provide object tangibility by associating the image-based representation with a physical prop (projecting the RTs) in a selective and flexible way. Overall, the proposed method provides a way to maintain an interactive frame rate with selective perceptual details in a large-scale virtual environment, while allowing the user to interact with virtual objects in a tangible way.

RDTC Optimized Compression of Image-Based Scene Representations (Part II): Practical Coding

Interactive streaming of compressed image-based scene representations requires random access to the reference image data. The degree of interframe dependencies exploited during encoding has an impact on the transmission and decoding time and, at the same time, delimits the (storage) rate-distortion (RD) tradeoff that can be achieved. The transmission data rate and the decoding complexity at the client have received attention in the literature, but their incorporation into the optimization procedure for compression and streaming is missing. If scenario-specific measures are considered, the traditional RD optimization can be extended to a tradeoff between the (storage) rate (R), distortion (D), transmission data rate (T), and decoding complexity (C). In the first part of this sequel of papers, we have theoretically analyzed the RDTC space for the compression of densely sampled image-based scene representations. In this second part, we consider practical RDTC optimization. We propose a modeling and encoding parameter selection procedure that allows us to adapt the compression to scenario-specific properties. The impact of client side caching is considered and evaluated using an experimental testbed. Our results show a significant reduction of the user perceived delay, memory consumption or required minimum channel and storage bitrate for RDTC optimized streams compared to classical RD optimized or independently encoded scene representations.

RDTC Optimized Compression of Image-Based Scene Representations (Part I): Modeling and Theoretical Analysis

Rendering of virtual views in interactive streaming of compressed image-based scene representations requires random access to arbitrary parts of the reference image data. The degree of interframe dependencies exploited during encoding has an impact on the transmission and decoding time and, at the same time, delimits the (storage) rate-distortion (RD) tradeoff that can be achieved. In this work, we extend the classical RD optimization approach using hybrid video coding concepts to a tradeoff between the storage rate (R), distortion (D), transmission data rate (T), and decoding complexity (C). We present a theoretical model for this RDTC space with a focus on the decoding complexity and, in addition, the impact of client side caching on the RDTC measures is considered and evaluated. Experimental results qualitatively match those predicted by our theoretical models and show that an adaptation of the encoding process to scenario specific parameters like computational power of the receiver and channel throughput can significantly reduce the user-perceived delay or required storage for RDTC optimized streams compared to RD optimized or independently encoded scene representations.

Photorealistic modeling of tissue reflectance properties

Objective: For Minimally Invasive Surgery (MIS) procedures, specular highlights constitute important visual cues for gauging tissue deformation as well as perceiving depth and orientation. This paper describes a novel reflectance modeling technique that is particularly suitable for simulating light interaction behavior with mucus-covered tissue surfaces. Methods: The complex and largely random tissue-light interaction behavior is modeled with a noise-based approach. In the proposed technique, Perlin noise is used to modulate the shape of specular highlights and imitate the effects of the complex tissue structure on reflected lighting. For efficient execution, the noise texture is generated in pre-processing and stored in an image-based representation, i.e., a reflectance map. At run-time, the graphics hardware is used to attain per-pixel control and achieve realistic tissue appearance. Results: The reflectance modeling technique has been used to replicate light-tissue reflection in surgical simulation. By comparing the results acquired against those obtained from conventional per-vertex Phong lighting and OpenGL multi-texturing, it is observed that the noise-based approach achieves improved tissue appearance similar to that observed in real procedures. Detailed user evaluation demonstrates the quality and practical value of the technique for increased perception of photorealism. Conclusion: The proposed technique presents a practical strategy for surface reflectance modeling that is suitable for real-time interactive surgical simulation. The use of graphics hardware further enhances the practical value of the technique.

Content-based retrieval of 3D models through curvature maps: a CBR approach exploiting media conversion

Multimedia digital libraries are gaining an increasing relevance, and are growing in both number and size. Moreover, advancements in acquisition and authoring technologies have also caused new media types to be added to these libraries. For instance, animations and 3D models have recently come alongside more traditional media such as text, images, audio and video. In particular, even if 3D objects were initially included in multimedia digital libraries in the form of images (i.e., views of the objects), they are now also available in the form of 3D models. These allow to overcome the inherent limitations of an image-based representation of a three-dimensional reality. In this paper we present a novel method for description and retrieval of 3D models that relies on curvature maps: after an initial pre-processing of the model, differential properties of points on the surface of the 3D object are evaluated; the model surface is then deformed into an ellipsoid, and is mapped onto a 2D image retaining curvature information of the original model. Through this media conversion from 3D model to image map, we enable application of analysis, description, and matching techniques which have been originally conceived for the image medium. In particular, the two approaches described in this paper rely on histograms for a coarse but efficient description of tiles of the curvature map, and on weighted walkthroughs for a more precise description of homogeneous regions of the map, taking into account their spatial arrangement and areas. Experimental results reveal that both techniques can be successfully applied to curvature maps. The two approaches have been compared against other solutions presented in literature, and have shown a good retrieval performance (particularly in the case of region-based description).

Three-dimensional natural video system based on layered representation of depth maps

In this paper, we propose a new three-dimensional (3-D) video system using a depth image-based representation (DIBR). In order to represent 3-D natural videos, we exploit texture images and synchronized depth maps, which are decomposed into four layers; number of layer (NOL), grid, boundary and feature point layers. With NOL, grid, and boundary layers, we define minimum required consecutive surfaces for 3-D natural videos. Then, we enhance their visual qualities by adding feature point layers. We regard NOL, grid and boundary layers as a base layer and feature point layers as enhancement layers to construct multi-layer representation of 3-D natural videos. With our proposed system, we cannot only represent depth maps efficiently, but also render 3-D natural videos in real time within reliable visual qualities. In addition, 3-D natural videos can be reconstructed adaptively according to consumers' capabilities and target applications within the framework of multi-layer representation. Experimental results have demonstrated that our proposed system can render and compress 3-D natural videos efficiently, as well as supporting the functionality of multi-view image generation

RAYSET: A TAXONOMY FOR IMAGE-BASED RENDERING

A new concept, referred to as rayset, is discussed in this paper. It is a parametric function consisting of two mapping relations. The first one maps from a parameter space to the ray space, while the second one maps from the parameter space to the attribute space. A taxonomy is proposed based on the rayset concept whereby scene representations and scene reconstruction techniques used in image-based rendering are individually classified. Existing image-based rendering techniques are surveyed under the proposed classification. The review shows that different image-based scene representations, such as multiple-center-of-projection image and concentric mosaics, can be cast as different kinds of raysets. Different scene reconstruction approaches can be regarded as attempts to render different raysets. The concepts of rayset warping and rayset editing are also formulated. Under the rayset taxonomy, both techniques try to alter one of the mapping relations defined by a rayset without changing the other one.

3D Video Player System with Haptic Interaction based on Depth Image-Based Representation

In this paper, we propose a three-dimensional (3D) video player system that provides haptic interaction with objects in a video scene based on a depth image-based representation. In order to represent dynamic 3D scenes, 3D video media combining general color video and synchronized depth video (depth image sequences) containing per-pixel depth information were exploited. With the proposed system, viewers can touch and explore the shape of objects visible in a scene via a three degree-of-freedom (DOF) haptic interface. To support haptic interaction in the 3D video player, a modified proxy graph algorithm was proposed which calculates the contact force between the 3D video scene and the viewers' finger. In addition, the proposed system architecture was implemented which encompassed 3D video contents generation, depth data enhancement, compression, and haptic rendering. The proposed 3D video player system showed stable, smooth, and immersive interaction in real-time.

Photorealistic modeling of tissue reflectance properties

Objective: For Minimally Invasive Surgery (MIS) procedures, specular highlights constitute important visual cues for gauging tissue deformation as well as perceiving depth and orientation. This paper describes a novel reflectance modeling technique that is particularly suitable for simulating light interaction behavior with mucus-covered tissue surfaces. Methods: The complex and largely random tissue-light interaction behavior is modeled with a noise-based approach. In the proposed technique, Perlin noise is used to modulate the shape of specular highlights and imitate the effects of the complex tissue structure on reflected lighting. For efficient execution, the noise texture is generated in pre-processing and stored in an image-based representation, i.e., a reflectance map. At run-time, the graphics hardware is used to attain per-pixel control and achieve realistic tissue appearance. Results: The reflectance modeling technique has been used to replicate light-tissue reflection in surgical simulation. By comparing the results acquired against those obtained from conventional per-vertex Phong lighting and OpenGL multi-texturing, it is observed that the noise-based approach achieves improved tissue appearance similar to that observed in real procedures. Detailed user evaluation demonstrates the quality and practical value of the technique for increased perception of photorealism. Conclusion: The proposed technique presents a practical strategy for surface reflectance modeling that is suitable for real-time interactive surgical simulation. The use of graphics hardware further enhances the practical value of the technique.

The plenoptic video

This paper presents a system for capturing and rendering a dynamic image-based representation called the plenoptic video. It is a simplified light field for dynamic environments, where user viewpoints are constrained to the camera plane of a linear array of video cameras. Important issues such as multiple camera calibration, real-time compression, decompression and rendering are addressed. The system consists of a camera array of eight Sony CCX-Z11 CCD cameras and eight Pentium 4 1.8-GHz computers connected together through a 100 Base-T local area network. It is possible to perform software-assisted real-time MPEG-2 compression at a resolution of (720/spl times/480). Using selective transmission, we are able to stream continuously plenoptic video with (256/spl times/256) resolution at a rate of 15 f/s over the network. For rendering from raw data on the hard disk, real-time rendering can be achieved with a resolution of (720/spl times/480) and a rate of 15 f/s. A new compression algorithm using both temporal and spatial predictions is also proposed for the efficient compression of the plenoptic videos. Experimental results demonstrate the usefulness of the proposed parallel processing based system in capturing and rendering high-quality dynamic image-based representations using off-the-shelf equipment, and its potential applications in visualization and immersive television systems.

Skin Texture Modeling

Quantitative characterization of skin appearance is an important but difficult task. The skin surface is a detailed landscape, with complex geometry and local optical properties. In addition, skin features depend on many variables such as body location (e.g. forehead, cheek), subject parameters (age, gender) and imaging parameters (lighting, camera). As with many real world surfaces, skin appearance is strongly affected by the direction from which it is viewed and illuminated. Computational modeling of skin texture has potential uses in many applications including realistic rendering for computer graphics, robust face models for computer vision, computer-assisted diagnosis for dermatology, topical drug efficacy testing for the pharmaceutical industry and quantitative comparison for consumer products. In this work we present models and measurements of skin texture with an emphasis on faces. We develop two models for use in skin texture recognition. Both models are image-based representations of skin appearance that are suitably descriptive without the need for prohibitively complex physics-based skin models. Our models take into account the varied appearance of the skin with changes in illumination and viewing direction. We also present a new face texture database comprised of more than 2400 images corresponding to 20 human faces, 4 locations on each face (forehead, cheek, chin and nose) and 32 combinations of imaging angles. The complete database is made publicly available for further research.

Rendering Deformable Surface Reflectance Fields

Animation of photorealistic computer graphics models is an important goal for many applications. Image-based modeling has emerged as a promising approach to capture and visualize real-world objects. Animating image-based models, however, is still a largely unsolved problem. In this paper, we extend a popular image-based representation called surface reflectance field to animate and render deformable real-world objects under arbitrary illumination. Deforming the surface reflectance field is achieved by modifying the underlying impostor geometry. We augment the impostor by a local parameterization that allows the correct evaluation of acquired reflectance images, preserving the original light model on the deformed surface. We present a deferred shading scheme to handle the increased amount of data involved in shading the deformable surface reflectance field. We show animations of various objects that were acquired with 3D photography.

Depth Image-Based Representation and Compression for Static and Animated 3-D Objects

This paper describes a new family of three-dimensional (3-D) representations for computer graphics and animation, called depth image-based representations (DIBR), which have been adopted into MPEG-4 Part16: Animation Framework eXtension (AFX). Idea of the approach is to build a compact and photorealistic representation of a 3-D object or scene without using polygonal mesh. Instead, images accompanied by depth values for each pixel are used. This type of representation allows us to build and render novel views of objects and scene with an interactive rate. There are many different methods for the image-based rendering with depths, and the DIBR format is designed to efficiently represent the information necessary for such methods. The main formats of the DIBR family are SimpleTexture (an image together with depth array), PointTexture (an image with multiple pixels along each line of sight), and OctreeImage (octree-like data structure together with a set of images containing viewport parameters). In order to store and transmit the DIBR object, we develop a compression algorithm and bitstream format for OctreeImage representation.

Object-based and image-based object representations

An overview is presented of object-based and image-based representations of objects by their interiors. The representations are distinguished by the manner in which they can be used to answer two fundamental queries in database applications: (1) Feature query: given an object, determine its constituent cells (i.e., their locations in space). (2) Location query: given a cell (i.e., a location in space), determine the identity of the object (or objects) of which it is a member as well as the remaining constituent cells of the object (or objects). Regardless of the representation that is used, the generation of responses to the feature and location queries is facilitated by building an index (i.e., the result of a sort) either on the objects or on their locations in space, and implementing it using an access structure that correlates the objects with the locations. Assuming the presence of an access structure, implicit (i.e., image-based) representations are described that are good for finding the objects associated with a particular location or cell (i.e., the location query), while requiring that all cells be examined when determining the locations associated with a particular object (i.e., the feature query). In contrast, explicit (i.e., object-based) representations are good for the feature query, while requiring that all objects be examined when trying to respond to the location query. The goal is to be able to answer both types of queries with one representation and without possibly having to examine every cell. Representations are presented that achieve this goal by imposing containment hierarchies on either space (i.e., the cells in the space in which the objects are found), or objects. In the former case, space is aggregated into successively larger-sized chunks (i.e., blocks), while in the latter, objects are aggregated into successively larger groups (in terms of the number of objects that they contain). The former is applicable to image-based interior-based representations of which the space pyramid is an example. The latter is applicable to object-based interior-based representations of which the R-tree is an example. The actual mechanics of many of these representations are demonstrated in the VASCO JAVA applets found at http://www.cs.umd.edu/˜hjs/quadtree/index.html.