Free-viewpoint System Research Articles

Optimal Representations for Adaptive Streaming in Interactive Multiview Video Systems

Interactive multiview video streaming (IMVS) services permit to remotely navigate within a 3D scene with an immersive experience. This is possible by transmitting a set of reference camera views (anchor views), which are used by the clients to freely navigate in the scene and possibly synthesize additional viewpoints of interest. From a networking perspective, the big challenge in IMVS systems is to deliver to each client the best set of anchor views that maximizes the navigation quality, minimizes the view-switching delay and yet satisfies the network constraints. Integrating adaptive streaming solutions in free-viewpoint systems offers a promising solution to deploy IMVS in large and heterogeneous scenarios, as long as the multiview video representations on the server are properly selected. Therefore, we propose to optimize the multiview data at the server by minimizing the overall resource requirements while offering a good navigation quality to the different users. We propose a representation set optimization problem for multiview adaptive streaming systems, and we show that it is NP-hard. Therefore, we introduce the concept of multiview navigation segment that permits to cast the video representation set selection as an integer linear programming problem with a bounded computational complexity. We then show that the proposed solution reduces the computational complexity, while preserving optimality in most of the 3D scenes. We finally provide simulation results for different classes of users and show the gain offered by an optimal multiview video representation selection compared to recommended representation sets (e.g., Netflix and Apple ones) or to a baseline representation selection algorithm, where the encoding parameters are decided a priori for all the camera views.

建設機械の遠隔操作のための運動立体視を実現する高速・低遅延な自由視点システム

建設機械の遠隔操作のための運動立体視を実現する高速・低遅延な自由視点システム

Tuned depth signal analysis on merged transform domain for view synthesis in free viewpoint systems

Completely embedded in the 3D era, depth maps coding becomes a must in order to favour 3D admission to different fields of application, ranging from video games to medical imaging. This study presents a novel depth coding approach that, after a decimation step favouring the foreground, decomposes depth maps onto a set of sparse coefficients and redundant mixed discrete cosine and B-splines atoms highly correlated to depth maps piece-wise linear nature. Depth decomposition searches the best rate/distortion tradeoff through minimisation of an adaptive cost function, where its weight parameter is manipulated according to depth homogeneity. The bigger the parameter is, the more the sparsity is favoured at the expense of synthesis quality. Furthermore, handled distortion measure of the cost function quantifies the effect of depth maps coding on rendered views quality. The experiments show the relevance of the proposed method, able to obtain considerable tradeoffs between bitrate and synthesised views distortion.

A perceptive remote-control 3D TV interface

3D TV is a turning point in the history of television. Here, technologies from computer graphics, computer vision, multimedia, telecommunications, broadcasting, and other related fields merge to expand the sensations provided by watching conventional 2D TV.1 As a result, there has been much research on multi-view image processing and 3D display.2, 3 However, most of the previous studies have focused on acquiring and displaying images while optimizing their ‘naturalness’, simplicity of computation, and real-time operation. Free-viewpoint TV (FTV)4 takes this idea forward. Free-viewpoint systems allow a viewer to look at a scene from the perspective that they want rather than what a director chooses to provide.1 In order to generate a scene according to the user’s requirements, a position in 3D space has to be defined as the view point. To achieve this, we suggest an intelligent approach that automatically generates reference viewpoints based on a theory of human visual attention.Many studies of visual attention and eye movements have shown that humans generally only attend to a few areas in an image rather than scanning the whole, and visual attention models provide a general approach to controlling the activities of active vision systems.5 These models of selective visual attention have been suggested on the basis of evidence in psychology, psychophysics, physiology, etc., and can also be exploited for our purposes. Figure 1 shows an examples of how our artificial human visual attention system can be used to obtain a basis position to generate a scene that corresponds to a user’s viewpoint. The system also simultaneously computes the intensity of salience of an object in a given region. In fact, several objects or regions can be detected in terms of their their individual saliencies, which can in turn be used to describe them. The system computes early visual features from a set of pre-attentive feature maps in a massively parallel way. Activity from all feature maps is combined Figure 1. Schematic diagram showing the proposed system based on a bottom-up approach to attention.