Planar Video Research Articles

Stereo Vision Conversion from Planar Videos Based on Temporal Multiplane Images

With the rapid development of 3D movie and light-field displays, there is a growing demand for stereo videos. However, generating high-quality stereo videos from planar videos remains a challenging task. Traditional depth-image-based rendering techniques struggle to effectively handle the problem of occlusion exposure, which occurs when the occluded contents become visible in other views. Recently, the single-view multiplane images (MPI) representation has shown promising performance for planar video stereoscopy. However, the MPI still lacks real details that are occluded in the current frame, resulting in blurry artifacts in occlusion exposure regions. In fact, planar videos can leverage complementary information from adjacent frames to predict a more complete scene representation for the current frame. Therefore, this paper extends the MPI from still frames to the temporal domain, introducing the temporal MPI (TMPI). By extracting complementary information from adjacent frames based on optical flow guidance, obscured regions in the current frame can be effectively repaired. Additionally, a new module called masked optical flow warping (MOFW) is introduced to improve the propagation of pixels along optical flow trajectories. Experimental results demonstrate that the proposed method can generate high-quality stereoscopic or light-field videos from a single view and reproduce better occluded details than other state-of-the-art (SOTA) methods. https://github.com/Dio3ding/TMPI

STC: FoV Tracking Enabled High-Quality 16K VR Video Streaming on Mobile Platforms

The ultra-high-definition 16K Virtual Reality (VR) video is coming to ages with more ”real” virtual experience and less cybersickness. However, the huge bitrate and decoding overhead would overwhelm today’s network and mobile hardware. The widely-known Field-of-View (FoV) adaptation streaming method still has severe bitrate wastes and decoding overhead as it delivers FoV areas with grid-like static tiles. Inspired by this, we present a novel ShiftTile-traCking (STC) streaming scheme, which crops and delivers tiles by tracking FoV movement. It is equivalent to deliver an FoV planar video instead of VR videos. This would save huge bit-rate and reduce decoding complexity. We mainly entail three contributions. 1) To reduce projection distortions, a novel FoV-centric sphere projection is proposed, which projects VR videos with the center of users’ FoVs. 2) To cover diverse FoV movement trajectories with a limited number of tiles, we propose an optimal tiling algorithm by trajectory clustering. 3) To be resilient to FoV prediction errors, we propose an accuracy-sensitive streaming algorithm, which scales FoV areas by the prediction accuracy. The evaluation shows that under the same network conditions, STC improves up to 1.3dB V-PSNR, reduces up to 13.2% buffering ratio, and achieves 60% faster decoding speed (61.5 frames per second) compared with state-of-the-art solutions.

Comparison of three methods for reconstructing 3D motion from 2D video recordings for low cost gait analysis systems

Abstract Purpose: The aim of this study was to quantify the accuracy of 3D trajectory reconstructions performed from two planar video recordings, using three different reconstruction methods. Additionally, the recordings were carried out using easily available equipment, like built-in cellphone cameras, making the methods suitable for low-cost applications. Methods: A setup for 3D motion tracking was constructed and used to acquire 2D video recordings subsequently used to reconstruct the 3D trajectories by 1) merging appropriate coordinates, 2) merging coordinates with proportional scaling, and 3) calculating the 3D position based on markers’ projections on the viewing plane. As experimental verification, two markers moving at a fixed distance of 98.9 cm were used to assess the consistency of results. Next, gait analysis in five volunteers was carried out to quantify the differences resulting from different reconstruction methods. Results: Quantitative evaluation of the investigated 3D trajectories reconstruction methods showed significant differences between those methods, with the worst reconstruction approach resulting in a maximum error of 50% (standard deviation 13%), while the best resulting in a maximum error of 1% (standard deviation 0.44%). The gait analysis results showed differences in mean angles obtained with each reconstruction method reaching only 2°, which can be attributed to the limited measurement volume. Conclusions: Reconstructing 3D trajectory from 2D views without accounting for the “perspective error” results in significant reconstruction errors. The third method described in this study enables a significant reduction of this issue. Combined with the proposed setup, it provides a functional, low-cost gait analysis system.

A Fast Intra Prediction Algorithm Based on WMSE for 360-Degree Video

With the increased demand for virtual reality, 360-degree video coding becomes challenging. A 360-degree video should be projected as a planar video before encoding, but this process causes distortion whose degree depends on the latitude. Traditional coding algorithms cannot effectively adapt to this feature. In this study, a fast intra prediction algorithm based on WMSE for 360-degree video is proposed. The similarity between the mother block and child blocks can reflect whether the mother block needs to be divided. Spherical weights are introduced in WMSE, and the similarity expressed by WMSE can distinguish the video of different latitude, which is consistent with the weight characteristics of ERP projection. Experimental results show that the proposed algorithm achieves 32% time reduction, with only 0.29% luma Bjontegaard delta rate (BD-rate) increase on average. The proposed algorithm provides a more effective solution to reduce the coding complexity of 360-degree video.

A Fast Sample Adaptive Offset Algorithm for 360∘ Video

The increased demand for virtual reality brings more challenges to 360∘ video coding. 360∘ video needs to be projected as a planar video before encoding. This process will produce projection distortion. The degree of projection distortion depends on the location of the pixel. Traditional coding algorithms cannot respond to this feature efficiently enough. In this paper, a fast Sample Adaptive Offset (SAO) algorithm for 360∘ video is proposed. The proposed algorithm improves the SAO process. On the basis of retaining the whole SAO process, a simplified SAO process is added. First, the coding tree unit (CTU) of performing the simplified SAO process is filtered according to the rate distortion cost (RD-cost) of the intra- or inter-prediction and the location of the CTU. Subsequently, the CTU is sampled at intervals according to the equirectangular projection (ERP) characteristics, and the CTU of performing the simplified SAO process is determined. Experimental results show that the proposed algorithm achieves 60% time of SAO process reduction, with only 0.29% luma Bjontegaard delta rate (BD-rate) increases on average.

Audio-visual speech recognition integrating 3D lip information obtained from the Kinect

Audio-visual speech recognition (AVSR) has shown impressive improvements over audio-only speech recognition in the presence of acoustic noise. However, the problems of region-of-interest detection and feature extraction may influence the recognition performance due to the visual speech information obtained typically from planar video data. In this paper, we deviate from the traditional visual speech information and propose an AVSR system integrating 3D lip information. The Microsoft Kinect multi-sensory device was adopted for data collection. The different feature extraction and selection algorithms were applied to planar images and 3D lip information, so as to fuse the planar images and 3D lip feature into the visual-3D lip joint feature. For automatic speech recognition (ASR), the fusion methods were investigated and the audio-visual speech information was integrated into a state-synchronous two stream Hidden Markov Model. The experimental results demonstrated that our AVSR system integrating 3D lip information improved the recognition performance of traditional ASR and AVSR system in acoustic noise environments.

Indirect estimation of cable tension during gymnastic movements on rings

Knowledge of external forces acting on an athlete is often required to study performance and injury mechanisms, but direct measurements are difficult to obtain. While transducers in the rings cables typically measure external forces experienced by gymnasts performing static balances and swinging movements on rings, this solution is not always acceptable and an accurate technique that does not hinder the gymnasts’ performances is desirable. This study evaluates a video-based technique for estimating cable tension on the rings apparatus. Static loading of a rings frame coupled with a planar video analysis of the mandatory damped elastic devices (DEDs) provided a tension-deflection calibration curve. Three-dimensional video and cable tension data were obtained for two male gymnasts performing static balances and dynamic swinging movements. Cable tension was measured using a force link while a planar video analysis determined time histories of the DED deflections. Combined cable tension time histories were estimated fromrelative DED deflections coupled with the tension-deflection calibration curve and the orientation of the rings cables, and were compared to measured values to evaluate the proposed technique. For static balances the RMS difference between measured and estimated combined cable tension was 83 N. For dynamic swinging movements the maximum RMS difference was 189 N, equivalent to a 4% difference when expressed as a percentage of the peak value. The indirect video-based technique developed in this study accurately estimates combined cable tension throughout movements on rings and may be considered for studies where a remote measurement is required. There is potential for the technique to be extended to other sporting situations, where indirect accurate estimates of external forces acting on an athlete are needed.

Influence of Fulcrum Position on Springboard Response and Takeoff Performance in the Running Approach

The running forward springboard approaches of 20 divers on the U.S. senior national diving team were analyzed using planar video analysis and accelerometer output; the purpose was to investigate kinematic responses of the springboard tip and divers' lower extremities to alterations in fulcrum setting. Divers executed jump takeoffs from a 1-m springboard with the fulcrum at their preferred setting as well as 0.144 m (two fulcrum numbers) forward and 0.144 m back from that location. Potential advantages of setting the fulcrum back further from the tip (as opposed to closer) included greater downward board tip vertical velocity at the beginning of takeoff, more time to generate angular momentum, and increased vertical velocity going into flight. However, setting the fulcrum further back required the diver to achieve longer hurdle flight durations and to reverse downward motion from a more flexed knee position. Accelerometer output in the absence of video was not beneficial as a means of providing immediate performance feedback for coaching purposes.

A low-cost 20-22 GHz MIC active receiver/radiometer

A microwave integrated circuit active receiver is built and tested at 19-25 GHz. The receiver consists of a planar CPW-fed double folded-slot antenna coupled to a six-stage MESFET amplifier and followed by a planar Schottky-diode detector. The folded-slot antenna on a GaAs half-space results in a wide frequency bandwidth suitable for MMIC amplifiers. The measured system performance show a video responsivity close to 1 GV/W at 20 GHz with a 3-dB bandwidth of 1500 MHz. A novel method which uses the planar video detector after the amplifier stages as an RF mixer is used to measure the noise-figure of the direct detection radiometer. The system noise figure is 4.8 dB at 22 GHz. The radiometer sensitivity to a hot/cold load is 3.8 /spl mu/V/K. The measured antenna patterns show a 90% Gaussicity at 20-22 GHz. The active MIC receiver can be integrated monolithically for low-cost applications and is well suited for millimeter-wave linear imaging arrays.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Microscopic Structure of Turbulent Diffusion Flames.

Microscopic structure of turbulent diffusion flames is studied by a time-resolved planar video picture system obtained by a laser Rayleigh scattering (LRS) method and a single-point LRS measurement. The microscopic temperature is measured by using a two-dimensional LRS signal and image processing. Coaxial turbulent diffusion flames at moderate Reynolds numbers, which exhibit typical diffusion flame structures, are formed on laboratory-scale burners. It is found that the flame can be divided into four typical regions characterized by the distribution of macroscale of temperature fluctuations. These four regions are visualized by the two-dimensional LRS pictures. The turbulent heat transfer mechanisms in these four regions are discussed in terms of the two-dimensional LRS and the power spectral density of temperature fluctuations, in terms of one-point LRS. A cluster of temperature inhomogeneity is observed by cluster analysis in Regions I and III. It is found that different structures of microscopic inhomogeneity in the mixing region of Taylor's dissipation length scale appear, corresponding to the characteristics of mixing and combustion mechanisms in each region.