Monoscopic Video Research Articles

Stereoscopic depth perception through foliage

Both humans and computational methods struggle to discriminate the depths of objects hidden beneath foliage. However, such discrimination becomes feasible when we combine computational optical synthetic aperture sensing with the human ability to fuse stereoscopic images. For object identification tasks, as required in search and rescue, wildlife observation, surveillance, and early wildfire detection, depth assists in differentiating true from false findings, such as people, animals, or vehicles vs. sun-heated patches at the ground level or in the tree crowns, or ground fires vs. tree trunks. We used video captured by a drone above dense woodland to test users’ ability to discriminate depth. We found that this is impossible when viewing monoscopic video and relying on motion parallax. The same was true with stereoscopic video because of the occlusions caused by foliage. However, when synthetic aperture sensing was used to reduce occlusions and disparity-scaled stereoscopic video was presented, whereas computational (stereoscopic matching) methods were unsuccessful, human observers successfully discriminated depth. This shows the potential of systems which exploit the synergy between computational methods and human vision to perform tasks that neither can perform alone.

HEVC Based Frame Interleaved Coding Technique for Stereo and Multi-View Videos

The standard HEVC codec and its extension for coding multiview videos, known as MV-HEVC, have proven to deliver improved visual quality compared to its predecessor, H.264/MPEG-4 AVC’s multiview extension, H.264-MVC, for the same frame resolution with up to 50% bitrate savings. MV-HEVC’s framework is similar to that of H.264-MVC, which uses a multi-layer coding approach. Hence, MV-HEVC would require all frames from other reference layers decoded prior to decoding a new layer. Thus, the multi-layer coding architecture would be a bottleneck when it comes to quicker frame streaming across different views. In this paper, an HEVC-based Frame Interleaved Stereo/Multiview Video Codec (HEVC-FISMVC) that uses a single layer encoding approach to encode stereo and multiview video sequences is presented. The frames of stereo or multiview video sequences are interleaved in such a way that encoding the resulting monoscopic video stream would maximize the exploitation of temporal, inter-view, and cross-view correlations and thus improving the overall coding efficiency. The coding performance of the proposed HEVC-FISMVC codec is assessed and compared with that of the standard MV-HEVC’s performance for three standard multi-view video sequences, namely: “Poznan_Street”, “Kendo” and “Newspaper1”. Experimental results show that the proposed codec provides more substantial coding gains than the anchor MV-HEVC for coding both stereo and multi-view video sequences.

Dynamic Viewport Selection-Based Prioritized Bitrate Adaptation for Tile-Based 360° Video Streaming

Interactive 360° remote video applications have seen booming advancements due to the proliferation of smart display devices that enable a truly immersive experience. Compared to regular monoscopic videos, 360° videos have different requirements related to content preparation, packaging, transmission, specialized viewing equipment, and display factors (e.g., brightness, contrast, delay, frame rate, resolution, image quality, etc. In addition, 360° video requires substantial network and computational resources, which are challenging to achieve with conventional transmission and rendering infrastructure. Viewport-adaptive streaming is a common way to ensure visual quality under limited bandwidth resources. However, identifying, extracting, and rendering the true viewport in response to drastic head rotations can adversely affect user experience. This paper proposes two dynamic viewport selection approaches, which adapt the streamed regions based on content complexity variations and positional information to ensure viewport availability and smooth visual angles for VR users. They incorporate content information as well as user head movement patterns to support tile-based prioritized 360° video streaming. Moreover, a practical, prioritized bitrate adaptation approach, which requests selected tiles at appropriate quality levels, is also proposed to reduce the impact of inefficient bandwidth utilization in the VR scene. Experimental evaluations under real 4G bandwidth logs demonstrate that the proposed solutions outperform the closest state-of-the-art algorithms across multiple performance metrics, i.e., viewport overlap, perceived quality levels, quality fluctuations, and viewport bandwidth utilization.

Sea state from ocean video with singular spectrum analysis and extended Kalman filter

A method for estimating key parameters of ocean waves (the dominant frequency and the significant wave height) from uncalibrated monoscopic video is proposed, based on temporal variation of the wave field, specifically time series of pixel intensities. The methodology tracks the principal component of the movement of water in the video, which we propose is associated with the dominant frequency of the ocean. To accomplish this, the singular spectrum analysis algorithm and the extended Kalman filter are used. Then, the shape of an empirical spectrum is used in order to translate the dominant frequency output into a significant wave height estimation.

Sea State Estimation from Uncalibrated, Monoscopic Video

Video of the ocean surface is used as a means for estimating the sea state. Time series of pixel intensity values are given as input to a method that uses the Kalman filter and the least squares approximate solution for estimating the uncalibrated video amplitude spectrum. A method is proposed for scaling this spectrum to metres with the use of an empirical model of the ocean. The significant wave height is estimated from the calibrated video amplitude spectrum. The results are tested against two sets of video data, and buoy measurements in both cases are solely used for indicating the true state. For significant wave height values between 0.5 and 3.6 m, the maximum observed value of root mean square error is 0.37 m and of mean absolute percentage error 16%.

Deep-learning based reconstruction of the stomach from monoscopic video data

Abstract For the gastroscopic examination of the stomach, the restricted field of view related to the „keyhole“-perspective of the endoscope is known to be a visual limitation. Thus, a panoramic extension can enlarge the field of vision, supports the endoscopist during the examination, and ensures that all of the inner stomach walls are visually inspected. To compute such a panorama of the stomach, knowledge about the geometry of the underlying structure is required. Structure from motion an approach to reconstruct the necessary information about the 3D-structure from monocular image sequences as provided by a gastroscope. We examine and evaluate an existing deep neuronal network for stereo reconstruction, in order to approximate the geometry of stomach parts from a set of consecutive acquired image pairs from gastroscopic videos.

HEVC Based Mixed-Resolution Stereo Video Codec

This paper presents a high efficiency video codec-based spatial mixed-resolution stereo video codec. The proposed codec applies a frame interleaving algorithm to reorder the stereo video frames into a single stream of monoscopic video. The challenge for mixed-resolution video coding is to enable the codec to encode frames with different frame resolutions. This issue is addressed by superimposing a low-resolution replica of the decoded I-frame on its respective decoded picture, where remaining space of the frame is set to zero. This significantly reduces the computation cost for finding the best match. The proposed codec’s reference frames structure is designed to efficiently exploit both temporal and inter-view correlations. The performance of the proposed codec is assessed using five standard multiview video data sets and benchmarked against that of the anchor and the state-of-the-art techniques. Results show that the proposed codec yields significantly higher coding performance compared with the anchor and state-of-the-art techniques.

The Evaluation of Learning Curves in Novice Laparoscopists: Incorporating Direct Visualization into the Simulation Training Program

Technological developments have led to resurgent interest in stereoscopic visualization for laparoscopy. Evidence suggests that trainees achieve proficiency on the McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS) in less total training time when using stereo visualization. Data from trainees using the Fundamentals of Laparoscopic Surgery (FLS) curriculum to train the MISTELS tasks indicate that the peg transfer task requires the greatest training time to reach proficiency. This task challenges basic hand‐eye coordination that is critical to performing more complex tasks. As monoscopic visualization remains the standard of care, our goal remains to train proficiency under monoscopic visualization. However, we propose incorporating stereo‐direct visualization (SDV) using open FLS box trainers, into the curriculum to accelerate learning. Novice surgery residents will train with the peg transfer task. Half will train to proficiency (< 54 sec) under standard monoscopic video visualization. The SDV group will train to proficiency, then switch to traditional monoscopic video visualization and continue to train until proficiency is reached. We hypothesize that SDV training will facilitate the perceptual switch to reach proficiency in less total time. Results from this study may be employed to further guide the development of laparoscopic training curricula.

Depth Transfer: Depth Extraction from Video Using Non-Parametric Sampling.

We describe a technique that automatically generates plausible depth maps from videos using non-parametric depth sampling. We demonstrate our technique in cases where past methods fail (non-translating cameras and dynamic scenes). Our technique is applicable to single images as well as videos. For videos, we use local motion cues to improve the inferred depth maps, while optical flow is used to ensure temporal depth consistency. For training and evaluation, we use a Kinect-based system to collect a large data set containing stereoscopic videos with known depths. We show that our depth estimation technique outperforms the state-of-the-art on benchmark databases. Our technique can be used to automatically convert a monoscopic video into stereo for 3D visualization, and we demonstrate this through a variety of visually pleasing results for indoor and outdoor scenes, including results from the feature film Charade.

An Automatic Method of Stereoscopic Video Synthesis from Monoscopic Video

This paper presents a novel depth-image-based rendering system for the generation of stereoscopic video from a monoscopic video sequence. The key point of the proposed system is that all the conversion procedures are accomplished automatically. First, we use Global Motion Estimation and an Adaptive Gaussian Mixture Model to segment the foreground and background. Second, different depth values are assigned to the foreground and background. Finally, image warping according to the depth map and stereoscopic video is synthesized. Experimental results show that the proposed method can effectively segment the foreground from the background and synthesize a satisfactory 3D effect, so the proposed approach is successful.

Multi-stream 3D video distribution over peer-to-peer networks

The recent advances in stereoscopic video capture, compression, and display have made 3-dimensional (3D) video a visually appealing and costly affordable technology. There have been a series of pioneer works on streaming 3D video over the Internet. Yet the remarkably increased data volume of 3D videos poses great challenges to the conventional client/server design, which has already suffered from supporting 2D videos.In this paper, we present an initial attempt toward efficient streaming of 3D videos over a peer-to-peer network. We show that the inherent multi-stream nature of 3D video makes playback synchronization more difficult, which is particularly acute with the existence of multiple senders in a peer-to-peer overlay. We address this by a novel 2-stream 2-stage buffer design, together with weighted data scheduling and light-weight synchronization. We further discuss a series of key practical issues toward implementing our peer-to-peer 3D video streaming system, including the weight modeling for data segments, the interactions with the RTP/RTCP protocol stack, and the inter-operability with monoscopic video as well as extension to multi-view video. We have evaluated the performance of our system under different end-system and network configurations with typical 3D video streams. The simulation results demonstrate the superiority of our system in terms of both scalability and streaming quality.

Content generation for 3D video/TV

A lack of suitable 3D content currently constitutes a major bottleneck for transferring the recent success of 3D cinema to our home TV. In this paper we describe a comprehensive 3D content generation system which enables to generate 3D content from either stereoscopic or monoscopic input videos. In particular, we present a method to convert original 2D image sequences to 3D content by adding depth information with only little user support. Furthermore, we show results of a stereo algorithm which provides the basis for automatic conversion of stereoscopic film material for viewing on different types of displays. In this context, we discuss the potential of inpainting techniques for filling in image regions that were originally occluded.

스테레오스코픽 비디오 방송 서비스를 위한 MPEG-2 전송스트림 구성 방안

본 논문은 3차원 입체효과를 나타내는 기술 중 가장 널리 쓰이고 있는 스테레오스코픽 영상을 방송 환경에서 활용되어지고 있는 MPEG-2 전송스트림(Transport Stream : TS)을 통해 효율적으로 서비스를 하기 위한 방법을 제안한다. 전통적인 MPEG-2 전송 스트림은 오디오, 비디오, 그리고 부가 데이터로 구성된 프로그램들을 각 데이터간의 동기 및 타이밍 기능을 제공하고 다중화함으로써 오디오, 비디오 및 데이터의 실시간 전송을 가능하게 한다. 그러나, 이러한 MPEG-2 전송스트림은 2D 영상을 기준으로 제정된 기술 규격으로 스테레오스코픽 영상을 위한 동기화, 다중화 처리 및 스테레오스코픽 동영상 식별이 어렵기에 스테레오스코픽 영상 서비스로 활용하기에는 어려움이 존재한다. 본 논문에서는 전통적인 MPEG-2 전송스트림의 2D 비디오 서비스와 호환성을 가지며 스테레오스코픽 비디오 서비스를 제공할 수 있도록 스테레오스코픽 비디오의 식별, 다중화 처리 및 동기화 처리를 위한 알고리듬을 제시하고 실제 구현을 통해 검증하였다. Stereoscopic video service is one of the most widely adopted technologies for 3-Dimensional technology. However, the stereoscopic video service has not been penetrated into the market since there does not exist a proper transmission method even though the conventional MPEG-2 Transport Stream (TS) has support for broadcast service of mono-scopic video contents. This is partly because the MPEG-2 TS does not support displaying a stereoscopic video content which consists of several composition types widely used in the market. This paper proposes the new MPEG-2 TS specification suitable for transmitting and displaying stereoscopic video contents. In order to show feasibility of the proposal, this paper describes an end-to-end system which is implemented with consideration of the method how to multiplex and de-multiplex stereoscopic video contents as well as keeping backward compatibility with the traditional MPEG-2 TS.

3D motion estimation for depth image coding in 3D video coding

In this paper, a new solution is introduced for the efficient compression of 3D video based on color and depth maps. While standard video codecs are designed for coding monoscopic videos, their application to depth maps is found to be suboptimal. With regard to the special properties of the depth maps, we propose an extension to conventional video coding in order to take advantage of object motion in the depth direction. Instead of performing a 2D motion search, as is common in conventional video codecs, we propose the use of a 3D motion search that is able to better exploit the temporal correlations of 3D content. In this new framework, the motion of blocks in depth maps are described using 3D motion vectors (x, y, z), representing the horizontal, vertical, and depth directions respectively. This leads to more accurate motion prediction and a smaller residual. The experimental results show that the proposed technique delivers an improvement in motion compensation, which leads to gains in compression efficiency.

3D motion estimation for depth information compression in 3D-TV applications

A new approach for predicting and coding depth information in 3D-TV (three-dimensional television) applications is presented. Properties of the depth information, which complements monoscopic video and enables 3D experience, are used in the proposed 3D motion prediction. The new approach leads to more efficient motion compensation and finally to higher compression. Built on the top of the conventional approaches for video coding, the proposed technique is suitable for integration in upcoming 3D-TV products.

3DTV over IP

In this paper, technologies for distribution of a variety of three-dimensional (3-D) video and 3-D TV services over IP networks are described. The approaches for encoding stereoscopic and MW have been standardized in the form of MPEG "video-plus-depth" and the JVT MVC standards. It has been shown that both approaches can encode stereoscopic video at about 1.2 times the bit rate of monoscopic video (when using unequal inter-view bit allocation in the case of MVC). Also this kind of interactive system increases the bandwidth requirement significantly. A multitude of strategies have been considered for streaming such encoded MW using RTP/UDP/IP or RTP/DCCP/IP. Video streaming architectures can be classified as 1) server unicasting to one or more clients, 2) server multicasting to several clients, 3) P2P unicast distribution, where each peer forwards packets to another peer, and 4) P2P multicasting, where each peer forwards packets to several other peers.

Geometrical recovery of missing data for shape from motion

This paper studies the geometrical recovery of an incomplete observation matrix for converting existing 2D video sequences to 3D content. In situations when converting previously recorded monoscopic video to 3D, several entries of the observation matrix have not been observed and other entries have been perturbed by the influence of noise. In such cases, there is no simple solution for SVD factorization for shape from motion. In this paper, a new recovery algorithm is proposed for recovering missing feature points, by minimizing the influence of noise, using iteratively geometrical correlations between a 2D observation matrix and 3D shape. Results in practical situations demonstrated with synthetic and real video sequences verify the efficiency and flexibility of the proposed method.

Accurate Estimation of Missing Data under Noise Distribution

3D contents have been becoming one of attractive multimedia and reality. In computer vision, 2D-to-3D conversion techniques require estimating missing data from noisy observations. When there is no missing data in the observation matrix, the accurate solution of such problem is known to be given by singular value decomposition (SVD). In the case of converting already recorded monoscopic video contents to 3D, several entries of the matrix have not been observed. Therefore, the problem has no simple solution, so it is necessary to estimate missing data. In this paper, we propose an estimation algorithm of missing data with minimizing the influence of noise embedded when tracking feature points from partial observations. The proposed method is an iterative affine SVD factorization method which can estimate the model parameters, given an incomplete set of the observation matrix. The main idea of our algorithm is to estimate missing data accurately even under noise distribution by using geometrical correlations between 2D and 3D error space. This paper consists of three main phases: geometrical correlations for estimating missing data, estimation algorithm, and analyzing the results for video sequences. The accurate results in practical situations as demonstrated here with synthetic and real video sequences show the efficiency and flexibility of the proposed method.

Stereoscopic video generation based on efficient layered structure and motion estimation from a monoscopic image sequence

This paper presents a novel object-based method for the generation of a stereoscopic image sequence from a monoscopic video, using bidirectional two-dimensional motion estimation for the recovery of rigid motion and structure and a Bayesian framework to handle occlusions. The latter is based on extended Kalman filters and an efficient method for reliably tracking object masks. Experimental results show that the layered object scene representation, combined with the proposed algorithm for reliably tracking object masks throughout the sequence, yields very accurate results.

Estimating Three-Dimensional Space from Multiple Two-Dimensional Views

The most common visual feedback technique in teleoperation is in the form of monoscopic video displays. As robotic autonomy increases and the human operator takes on the role of a supervisor, three-dimensional information is effectively presented by multiple, televised, two-dimensional (2-D) projections showing the same scene from different angles. To analyze how people go about using such segmented information for estimations about three-dimensional (3-D) space, 18 subjects were asked to determine the position of a stationary pointer in space; eye movements and reaction times (RTs) were recorded during a period when either two or three 2-D views were presented simultaneously, each showing the same scene from a different angle. The results revealed that subjects estimated 3-D space by using a simple algorithm of feature search. Eye movement analysis supported the conclusion that people can efficiently use multiple 2-D projections to make estimations about 3-D space without reconstructing the scene mentally in three dimensions. The major limiting factor on RT in such situations is the subjects' visual search performance, giving in this experiment a mean of 2270 msec (SD = 468; N = 18). This conclusion was supported by predictions of the Model Human Processor (Card, Moran, & Newell, 1983), which predicted a mean RT of 1820 msec given the general eye movement patterns observed. Single-subject analysis of the experimental data suggested further that in some cases people may base their judgments on a more elaborate 3-D mental model reconstructed from the available 2-D views. In such situations, RTs and visual search patterns closely resemble those found in the mental rotation paradigm (Just & Carpenter, 1976), giving RTs in the range of 5-10 sec.