Multi-view Content Research Articles

Color Correction for Immersive Video Applications

In this paper, the color correction method developed for immersive video systems is presented. The proposed method significantly increases the consistency of color characteristics of multiview sequences, understood both as the temporal and the inter-view consistency, what highly improves the subjective quality of the synthesized virtual views presented to the final user of the immersive video system. Moreover, the proposal allows to significantly increase the quality of the depth maps calculated for natural sequences, e.g., for views with colors inconsistent due to different lighting conditions. It enables more efficient compression of natural multiview video, as the newest encoding standards highly depend on the quality of depth maps. In order to evaluate the performance of the proposal, three experiments were conducted. In the first one, the proposal was compared to state of the art in the typical immersive video application – color correction of a natural multiview sequence. In the second experiment, the performance of the proposal was tested on the Middlebury stereo dataset. In both experiments, the quality of synthesized virtual views was assessed subjectively by a group of 70 naive viewers. The third experiment assessed the influence of color correction on the quality of estimated depth maps. All the experiments showed that the proposal significantly increases the color consistency of the multiview content. Due to the high usefulness and robustness, the proposed color correction method became the MPEG reference software for color correction. The implementation of the method is available for other researchers on the public repository.

No-Reference Depth Map Quality Evaluation Model Based on Depth Map Edge Confidence Measurement in Immersive Video Applications

When it comes to evaluating perceptual quality of digital media for overall quality of experience assessment in immersive video applications, typically two main approaches stand out: Subjective and objective quality evaluation. On one hand, subjective quality evaluation offers the best representation of perceived video quality assessed by the real viewers. On the other hand, it consumes a significant amount of time and effort, due to the involvement of real users with lengthy and laborious assessment procedures. Thus, it is essential that an objective quality evaluation model is developed. The speed-up advantage offered by an objective quality evaluation model, which can predict the quality of rendered virtual views based on the depth maps used in the rendering process, allows for faster quality assessments for immersive video applications. This is particularly important given the lack of a suitable reference or ground truth for comparing the available depth maps, especially when live content services are offered in those applications. This paper presents a no-reference depth map quality evaluation model based on a proposed depth map edge confidence measurement technique to assist with accurately estimating the quality of rendered (virtual) views in immersive multi-view video content. The model is applied for depth image-based rendering in multi-view video format, providing comparable evaluation results to those existing in the literature, and often exceeding their performance.

Feature Sampling Based Unsupervised Semantic Clustering for Real Web Multi-View Content

Real web datasets are often associated with multiple views such as long and short commentaries, users preference and so on. However, with the rapid growth of user generated texts, each view of the dataset has a large feature space and leads to the computational challenge during matrix decomposition process. In this paper, we propose a novel multi-view clustering algorithm based on the non-negative matrix factorization that attempts to use feature sampling strategy in order to reduce the complexity during the iteration process. In particular, our method exploits unsupervised semantic information in the learning process to capture the intrinsic similarity through a graph regularization. Moreover, we use Hilbert Schmidt Independence Criterion (HSIC) to explore the unsupervised semantic diversity information among multi-view contents of one web item. The overall objective is to minimize the loss function of multi-view non-negative matrix factorization that combines with an intra-semantic similarity graph regularizer and an inter-semantic diversity term. Compared with some state-of-the-art methods, we demonstrate the effectiveness of our proposed method on a large real-world dataset Doucom and the other three smaller datasets.

Interview

interview

Low-Complexity Texture Video Coding Based on Motion Homogeneity for 3D-HEVC

Three-dimensional extension of the high efficiency video coding (3D-HEVC) is an emerging international video compression standard for multiview video system applications. Similar to HEVC, a computationally expensive mode decision is performed using all depth levels and prediction modes to select the least rate-distortion (RD) cost for each coding unit (CU). In addition, new tools and intercomponent prediction techniques have been introduced to 3D-HEVC for improving the compression efficiency of the multiview texture videos. These techniques, despite achieving the highest texture video coding efficiency, involve extremely high-complex procedures, thus limiting 3D-HEVC encoders in practical applications. In this paper, a fast texture video coding method based on motion homogeneity is proposed to reduce 3D-HEVC computational complexity. Because the multiview texture videos instantly represent the same scene at the same time (considering that the optimal CU depth level and prediction modes are highly multiview content dependent), it is not efficient to use all depth levels and prediction modes in 3D-HEVC. The motion homogeneity model of a CU is first studied according to the motion vectors and prediction modes from the corresponding CUs. Based on this model, we present three efficient texture video coding approaches, such as the fast depth level range determination, early SKIP/Merge mode decision, and adaptive motion search range adjustment. Experimental results demonstrate that the proposed overall method can save 56.6% encoding time with only trivial coding efficiency degradation.

Robust Multiview Synthesis for Wide-Baseline Camera Arrays

In many advanced multimedia systems, multiview content can offer more immersion compared to classical stereoscopy. The feeling of immersiveness is increased substantially by offering motion-parallax, as well as stereopsis. This drives both the so-called free-navigation and super-multiview technologies. However, it is currently still challenging to acquire, store, process, and transmit this type of content. This paper presents a novel multiview-interpolation framework for wide-baseline camera arrays. The proposed method comprises several novel components, including point cloud-based filtering, improved de-ghosting, multireference color blending, and depth-aware MRF-based disocclusion inpainting. The method offers robustness against depth errors caused by quantization and smoothing across object boundaries. Furthermore, the available input color and depth are maximally exploited while preventing propagation of unreliable information to virtual viewpoints. The experimental results show that the proposed method outperforms the state-of-the-art View Synthesis Reference Software (VSRS 4.1) both in objective terms as well as subjectively, based on a visual assessment on a high-end light-field three-dimensional display.

Conversion of sparsely-captured light field into alias-free full-parallax multiview content

We propose shearlet decomposition based light field (LF) reconstruction and filtering techniques for mitigating artifacts in the visualized contents of 3D multiview displays. Using the LF reconstruction capability, we first obtain the densely sampled light field (DSLF) of the scene from a sparse set of view images. We design the filter via tiling the Fourier domain of epipolar image by shearlet atoms that are directionally and spatially localized versions of the desired display passband. In this way, it becomes possible to process the DSLF in a depth-dependent manner. That is, the problematic areas in the 3D scene that are outside of the display depth of field (DoF) can be selectively filtered without sacrificing high details in the areas near the display, i.e. inside the DoF. The proposed approach is tested on a synthetic scene and the improvements achieved by means of the quality of the visualized content are verified, where the visualization process is simulated using a geometrical optics model of the human eye.

Quality of Experience in a Stereoscopic Multiview Environment

In this paper, we investigate how visualization factors, such as disparity, mobility, angular resolution, and viewpoint interpolation, influence the quality of experience (QoE) in a stereoscopic multiview environment. In order to do so, we set up a dedicated testing room and conducted subjective experiments. We also developed a framework that emulates a supermultiview environment. This framework can be used to investigate and assess the effects of angular resolution and viewpoint interpolation on the QoE produced by multiview systems, and provide relevant cues as to how the baselines of cameras and interpolation strategies in such systems affect user experience. Aspects such as visual comfort, model fluidity, sense of immersion, and the three-dimensional (3D) experience as a whole have been assessed for several test cases. Obtained results suggest that user experience in an motion parallax environment is not as critically influenced by configuration parameters such as disparity as initially thought. In addition, extensive subjective tests have indicated that while users are very sensitive to angular resolution in multiview 3D systems, this sensitivity tends not to be as critical when a user is performing a task that involves a great amount of interaction with the multiview content. These tests have also indicated that interpolating intermediate viewpoints can be effective in reducing the required view density without degrading the perceived QoE.

Hole Filling for View Synthesis Using Depth Guided Global Optimization

View synthesis is an effective way to generate multi-view contents from a limited number of views, and can be utilized for 2-D-to-3-D video conversion, multi-view video compression, and virtual reality. In the view synthesis techniques, depth-image-based rendering (DIBR) is an important method to generate virtual view from video-plus-depth sequence. However, some holes might be produced in the DIBR process. Many hole filling methods have been proposed to tackle this issue, but most of them cannot achieve globally coherent or acquire trusted contents. In this paper, a hole filling method with depth-guided global optimization is proposed for view synthesis. The global optimization is achieved by iterating the spatio-temporal approximate nearest neighbor (ANN) search and video reconstruction step. Directly applying global optimization might introduce some foreground artifacts to the synthesized video. To prevent this problem, some strategies have been developed in this paper. The depth information is applied to guide the spatio-temporal ANN searching and the initialization step is specified in the global optimization procedure. Our experimental results have demonstrated that the proposed method has better performance compared with other methods in terms of visual quality, trusted textures, and temporal consistency in the synthesized video.

Low-complexity encoding scheme for multiview video content

On-site multiview video viewing system can provide people with more immersive viewing experience on various live video shows, such as sports and concerts. To address the high computational complexity of multi-view video coding, several researches have worked on fast motion estimation, fast disparity estimation, and/or fast mode decision. In this paper, we propose a low-complexity encoding scheme that can further exploit inter-view dependencies and depth information. A fast mode selection with threshold-based early termination is presented to select a minimal set of mode candidates according to the reference macroblocks of neighbouring views and expedite the mode decision process. Further, this study also demonstrates that disparity estimation computation and search ranges of both motion estimation and disparity estimation can be significantly refined by using inter-view dependencies and depth information. The experimental results show that the proposed encoding method reduces the coding time by 80% with a negligible PSNR loss.

3DTV at home

Stereoscopic 3D (S3D) movies have become widely popular in the movie theaters, but the adoption of S3D at home is low even though most TV sets support S3D. It is widely believed that S3D with glasses is not the right approach for the home. A much more appealing approach is to use automulti-scopic displays that provide a glasses-free 3D experience to multiple viewers. A technical challenge is the lack of native multiview content that is required to deliver a proper view of the scene for every viewpoint. Our approach takes advantage of the abundance of stereoscopic 3D movies. We propose a real-time system that can convert stereoscopic video to a high-quality multiview video that can be directly fed to automultiscopic displays. Our algorithm uses a wavelet-based decomposition of stereoscopic images with per-wavelet disparity estimation. A key to our solution lies in combining Lagrangian and Eulerian approaches for both the disparity estimation and novel view synthesis, which leverages the complementary advantages of both techniques. The solution preserves all the features of Eulerian methods, e.g., subpixel accuracy, high performance, robustness to ambiguous depth cases, and easy integration of inter-view aliasing while maintaining the advantages of Lagrangian approaches, e.g., robustness to large disparities and possibility of performing non-trivial disparity manipulations through both view extrapolation and interpolation. The method achieves real-time performance on current GPUs. Its design also enables an easy hardware implementation that is demonstrated using a field-programmable gate array. We analyze the visual quality and robustness of our technique on a number of synthetic and real-world examples. We also perform a user experiment which demonstrates benefits of the technique when compared to existing solutions.

Depth-reversal-free three-dimensional display by adjusting the light field alignment

A multi-view content generation method for three-dimensional (3D) display without depth-reversed area is demonstrated. The viewing zone periodicity of an autostereoscopic display is utilized to eliminate the depth-reversal by adjusting the light field alignment in the 3D light field. The conditions that a multi-view sequence should satisfies to form a depth-reversal-free sequence are given and detailed processes for converting the multi-view sequence into a depth-reversal-free sequence are presented. Experimental results show that the depth-reversed area is well eliminated, and the smooth motion parallax can be achieved when the viewer moves in front of the display within a 60°viewing angle. The number of presented perspectives can be 8 times more than the viewpoint number of the autostereoscopic display with the help of viewing zones integration.

Vivid-DIBR Based 2D–3D Image Conversion System for 3D Display

The multi-view perspectives are captured around a scene by camera array at the same time, and multi-view images cannot only keep two dimensions information but also has the depth architecture of a scene. Most of 3D displays support the multi-view contents, while there is a bottleneck of the 3D TV system to broadcast the larger storage and bandwidth for enduser. For saving bandwidth, a new TV broadcasting channel transfers third dimensional information, i.e., depth map has been utilized. The 3D display renders intermediate image and a depth map to form different views, as well as synthesizes these images into one image; this technology is the renowned Depth Image Based Rendering (DIBR). In other words, any 2D content can be transformed into multi-view images according to their corresponding depth map. Our system processes 2D images to capture a depth map of the intermediate view image during 3D contents capturing. In this paper, we utilize the vanishing point detection and color image segmentation technique to find the objects and deepest point (i.e., vanishing point) in the image, and then we assign depth value by comparing the vanishing point of object with image. After depth map generation, we propose the Vivid-DIBR system to imitate how human eyes see things. On the other hand, this system solves the holes (warping error points) problem by redistributing corresponding depth map. The design aims to convert 2D images into multi-view images which will be suitable for any interlacing 3D display by adjusting the focal plane location.

Transmission Policy Selection for Multi-View Content Delivery Over Bandwidth Constrained Channels.

I formulate an optimization framework for computing the transmission actions of streaming multi-view video content over bandwidth constrained channels. The optimization finds the schedule for sending the packetized data that maximizes the reconstruction quality of the content, for the given network bandwidth. Two prospective multi-view content representation formats are considered: 1) MVC and 2) video plus depth. In the case of each, I formulate directed graph models that characterize the interdependencies between the data units that comprise the content. For the video plus depth format, I develop a novel space-time error concealment strategy that reconstructs the missing content based on received data units from multiple views. I design multiple techniques to solve the optimization problem of interest, at varying degrees of complexity and accuracy. In conjunction, I derive spatiotemporal models of the reconstruction error for the multi-view content that I employ to reduce the computational requirements of the optimization. I study the performance of my framework via simulation experiments. Significant gains in terms of rate-distortion efficiency are demonstrated over various reference methods.

A Novel Approach Based on Multi-View Content Analysis and Semi-Supervised Enrichment for Movie Recommendation

Although many existing movie recommender systems have investigated recommendation based on information such as clicks and tags, much less efforts have been made to explore the multimedia content of movies, which has potential information for the elicitation of the user’s visual and musical preferences. In this paper, we explore the content from three media types (image, text, audio) and propose a novel multi-view semi-supervised movie recommendation method, which represents each media type as a view space for movies. The three views of movies are integrated to predict the rating values under the multi-view framework. Furthermore, our method considers the casual users who rate limited movies. The algorithm enriches the user profile with a semi-supervised way when there are only few rating histories. Experiments indicate that the multimedia content analysis reveals the user’s profile in a more comprehensive way. Different media types can be a complement to each other for movie recommendation. And the experimental results validate that our semi-supervised method can effectively enrich the user profile for recommendation with limited rating history.

Adaptive multiview video streaming: challenges and opportunities

Delivering multiview video content over present packet networks poses multiple challenges. First, the best effort nature of the Internet exposes media packets to variable bandwidth, loss, and delay as they traverse the network. Second, the prediction dependencies employed to maximize compression efficiency make the reconstruction process at the client extremely vulnerable to missing data. Third, the heterogeneity of client devices in terms of computing power, display capabilities, and access link capacity necessitates customizing the streaming process per user. My article reviews existing opportunities for addressing these challenges from within each of the three main stages of the content delivery pipeline (i.e., encoding, transmission, and reconstruction). Concretely, I first describe adaptive source coding techniques that construct a compressed representation of the multiview video source that exhibits resilience to network bandwidth variations and client view selection uncertainty. Then I discuss intelligent methods for error protection, caching, and packet scheduling that organize the transmission of multiview data in a bandwidth-effective way. Here, I also review prospective multipath and cloud-assisted techniques for multiview video streaming. Finally, I identify robust client-side content reconstruction schemes and adaptive media playout methods that can minimize the impact of missing data and enhance the user's interactive experience. Then I proceed to describe community-driven streaming techniques for delivering interactive multiview content over a population of social peers. The article concludes with an outline of approaches for synergistic exploitation of the techniques I will present theretofore, jointly across the different layers of the network protocol stack at which they individually operate. Here, I also highlight the main deployment challenges for some of these techniques, and how their design should be addressed accordingly, to overcome them.

Synthetic content generation for auto-stereoscopic displays

Due to the appearance of auto-stereoscopic visualization as one of the most emerging tendencies used in displays, new content generation techniques for this kind of visualization are required. In this paper we present a study for the generation of multi-view synthetic content, studying several camera setups (planar, cylindrical and hyperbolic) and their configurations. We discuss the different effects obtained varying the parameters of these setups. A study with several users was made to analyze visual perceptions, asking them for their optimal visualization. To create the virtual content, a multi-view system has been integrated in a powerful game engine, which allows us to use the latest graphics hardware advances. This integration is detailed and several demos and videos are attached with this paper, which represent a virtual world for auto-stereoscopic displays and the same scenario in a two-view anaglyph representation for being visualized in any conventional display. In all these demos, the parameters studied can be modified offering the possibility of easily appreciate their effects in a virtual scene.

Analysis of image distortion based on light ray field by multi-view and horizontal parallax only integral imaging display

Three-dimensional image distortion caused by mismatch between autostereoscopic displays and contents is analyzed. For a given three-dimensional object scene, the original light ray field in the contents and deformed one by the autostereoscopic displays are calculated. From the deformation of the light ray field, the distortion of the resultant three-dimensional image is finally deduced. The light ray field approach enables generalized distortion analysis across different autostereoscopic display techniques. The analysis result is verified experimentally when multi-view contents are applied to a multi-view display of non-matching parameters and a horizontal parallax only integral imaging display.

Content-adaptive parallax barriers

We optimize automultiscopic displays built by stacking a pair of modified LCD panels. To date, such dual-stacked LCDs have used heuristic parallax barriers for view-dependent imagery: the front LCD shows a fixed array of slits or pinholes, independent of the multi-view content. While prior works adapt the spacing between slits or pinholes, depending on viewer position, we show both layers can also be adapted to the multi-view content, increasing brightness and refresh rate. Unlike conventional barriers, both masks are allowed to exhibit non-binary opacities. It is shown that any 4D light field emitted by a dual-stacked LCD is the tensor product of two 2D masks. Thus, any pair of 1D masks only achieves a rank-1 approximation of a 2D light field. Temporal multiplexing of masks is shown to achieve higher-rank approximations. Non-negative matrix factorization (NMF) minimizes the weighted Euclidean distance between a target light field and that emitted by the display. Simulations and experiments characterize the resulting content-adaptive parallax barriers for low-rank light field approximation.

다시점 비디오 부호화를 위한 움직임벡터 예측값 선택 방법

본 논문에서는 JVT에서 표준화가 진행 중인 다시점 비디오 부호화의 부호화 효율을 높이기 위해 다시점 비디오의 예측구조의 특성을 고려하여 움직임벡터 예측값을 선택하는 방법을 제안한다. 다시점 비디오 부호화에서 시간 방향과 뷰 방향의 예측을 수행하면서 서로 다른 성향의 움직임벡터가 발생된다. 또한 주변 블록에서 시간 방향과 뷰 방향 참조의 움직임벡터가 혼재하는 상황에서 동일한 성향의 움직임벡터가 선택되지 않는 경우가 발생되어 움직임벡터의 예측 효율이 저하된다. 이에 다시점 비디오 부호화의 예측구조에 따라 움직임벡터 예측값을 선택할 수 있는 방법을 제안한다. 제안하는 움직임벡터 예측값 선택 방법을 통하여 다시점 비디오 부호화의 압축 효율을 높일 수 있었으며, JMVM 3.6에서 움직임벡터 예측값을 선택하는 방법을 사용하는 다시점 비디오 부호화와 비교하여 PSNR(Peak Signal-to-Noise Ratio)이 약<TEX>$0.03{\sim}0.1$</TEX> dB 정도 향상되었다. In this paper, we propose a method to select motion vector predictor by considering prediction structure of a multi view content for coding efficiency of multi view coding which is being standardized in JVT. Motion vector of a different tendency is happened while carrying out temporal and view reference prediction of multi-view video coding. Also, due to the phenomena of motion vectors being searched in both temporal and view order, the motion vectors do not agree with each other resulting a decline in coding efficiency. This paper is about how the motion vector predictor are selected with information of prediction structure. By using the proposed method, a compression ratio of the proposed method in multi-view video coding is increased, and finally <TEX>$0.03{\sim}0.1$</TEX> dB PSNR(Peak Signal-to-Noise Ratio) improvement was obtained compared with the case of JMVM 3.6 method.