Free-viewpoint Video Streaming Research Articles

Real-time Free-viewpoint Video Streaming Technology by Using 5G Wireless Network

来る5G時代の新たな映像体験技術として，ユーザに高い臨場感を提供する自由視点映像合成技術のニーズが高まっている．これまで，数多くの自由視点映像合成技術が提案されてきたが，複数の高精細なカメラ映像から被写体の3次元形状を推定するために膨大な時間を要するといった課題があり，コンテンツのオフライン制作が必須であった．この制約から，放送波や無線を用いた自由視点映像のリアルタイム配信はこれまで実現されてこなかった．本論文では，リアルタイム処理が可能な自由視点映像合成技術，および5G網を用いた配信システムを提案する．提案手法では，従来，点群として表現されていた被写体3次元形状を，平面によって表現することで高速な映像合成を可能としている．また，当該技術を含む配信システムに5G網を用いることで，ユーザ操作に対して遅れのない合成映像の提示を実現した．実際のスポーツシーンにおける5G網を用いた実証実験において，本提案システムの有効性を確認した．

Optimized camera handover scheme in free-viewpoint video streaming

Free-viewpoint video (FVV) is a promising approach that allows users to control their viewpoint and generate virtual views from any desired perspective. The individual user viewpoints are synthesized from two or more camera streams and correspondent depth sequences. In case of continuous viewpoint changes, the camera inputs of the view-synthesis process must be changed in a seamless way, to avoid the starvation of the viewpoint synthesizer algorithm. Starvation occurs when the desired user viewpoint cannot be synthesized with the currently streamed camera views, and thus, the FVV playout interrupts. In this paper, we propose three different camera handover schemes (TCC, MA, and SA) based on viewpoint prediction to minimize the probability of playout stalls and find the trade-off between the image quality and the camera handover frequency. Our simulation results show that the introduced camera switching methods can reduce the handover frequency with more than 40%, and hence, the viewpoint synthesis starvation and the playout interruption can be minimized. By providing seamless viewpoint changes, the quality of experience can be significantly improved, making the new FVV service more attractive in the future.

무선 자유시점 비디오 스트리밍에서 인기도 기반 전송 기기 선택 기법

무선 자유시점 비디오 스트리밍에서 인기도 기반 전송 기기 선택 기법

Anchor View Allocation for Collaborative Free Viewpoint Video Streaming

In free viewpoint video, a viewer can choose at will any camera angle or the so-called “virtual view” to observe a dynamic 3-D scene, enhancing his/her depth perception. The virtual view is synthesized using texture and depth videos of two anchor camera views via depth-image-based rendering (DIBR). We consider, for the first time, collaborative live streaming of a free viewpoint video, where a group of users may interactively pull and cooperatively share streams of different anchor views. There is a cost to access the anchor views from the live source, a cost to “reconfigure” the peer network due to a change in selected anchors during view switching, and a distortion cost due to the distance of the virtual views to the received anchor views at users. We optimize the anchor views allocated to users so as to minimize the overall streaming cost given by the access cost, reconfiguration cost, and view distortion cost. We first show that, if the reconfiguration cost due to view switching is negligible, the view allocation problem can be optimally and efficiently solved in polynomial time using dynamic programming. For the case of non-negligible reconfiguration cost, the problem becomes NP-hard. We thus present a locally optimal and centralized algorithm inspired by Lloyd's algorithm used in non-uniform scalar quantization. We further propose a distributed algorithm with convergence guarantee, where each peer group independently makes merge-and-split decisions with a well-defined fairness criteria. Simulation results show that our algorithms achieve low streaming cost due to its excellent anchor view allocation.