Stereoscopic Content Research Articles

Stereoscopic video deblurring transformer

Stereoscopic cameras, such as those in mobile phones and various recent intelligent systems, are becoming increasingly common. Multiple variables can impact the stereo video quality, e.g., blur distortion due to camera/object movement. Monocular image/video deblurring is a mature research field, while there is limited research on stereoscopic content deblurring. This paper introduces a new Transformer-based stereo video deblurring framework with two crucial new parts: a self-attention layer and a feed-forward layer that realizes and aligns the correlation among various video frames. The traditional fully connected (FC) self-attention layer fails to utilize data locality effectively, as it depends on linear layers for calculating attention maps The Vision Transformer, on the other hand, also has this limitation, as it takes image patches as inputs to model global spatial information. 3D convolutional neural networks (3D CNNs) process successive frames to correct motion blur in the stereo video. Besides, our method uses other stereo-viewpoint information to assist deblurring. The parallax attention module (PAM) is significantly improved to combine the stereo and cross-view information for more deblurring. An extensive ablation study validates that our method efficiently deblurs the stereo videos based on the experiments on two publicly available stereo video datasets. Experimental results of our approach demonstrate state-of-the-art performance compared to the image and video deblurring techniques by a large margin.

A new mixed reality tool for training in minimally invasive robotic-assisted surgery

Robotic-assisted surgery (RAS) is developing an increasing role in surgical practice. Therefore, it is of the utmost importance to introduce this paradigm into surgical training programs. However, the steep learning curve of RAS remains a problem that hinders the development and widespread use of this surgical paradigm. For this reason, it is important to be able to train surgeons in the use of RAS procedures. RAS involves distinctive features that makes its learning different to other minimally invasive surgical procedures. One of these features is that the surgeons operate using a stereoscopic console. Therefore, it is necessary to perform RAS training stereoscopically. This article presents a mixed-reality (MR) tool for the stereoscopic visualization, annotation and collaborative display of RAS surgical procedures. The tool is an MR application because it can display real stereoscopic content and augment it with virtual elements (annotations) properly registered in 3D and tracked over time. This new tool allows the registration of surgical procedures, teachers (experts) and students (trainees), so that the teacher can share a set of videos with their students, annotate them with virtual information and use a shared virtual pointer with the students. The students can visualize the videos within a web environment using their personal mobile phones or a desktop stereo system. The use of the tool has been assessed by a group of 15 surgeons during a robotic-surgery master’s course. The results show that surgeons consider that this tool can be very useful in RAS training.

Stereo image quality assessment using deformed pixels and Sobel magnitude

The application of 3D technology is rapidly expanding, and stereoscopic imagery is typically used to display 3D data. However, compression, transmission, and other necessary processes may reduce the quality of these images. Stereo image quality assessment (SIQA) has gained more attention to guarantee that customers have a positive watching experience. In order to provide the highest level of experience, it is necessary to develop a quality evaluation mechanism for stereoscopic content that is both dependable and precise. A full-reference method for SIQA is presented in this paper. Compared to previous measures, this method gives users more freedom to use distorted pixel metrics and edge similarity. The binocular summation map is calculated by adding the left and right images for a stereo pair. Improved gradient similarity based distorted pixel measure (SGSDM) is used to calculate the quality of binocular summation. The scored 3D LIVE IQA database is used to evaluate the correlation of the proposed metric with the DMOS subjective score given by the database. The proposed method’s efficacy is demonstrated by experimental comparisons.

Incidence of stereo blindness in a recent VR distance perception user study

Estimates of stereo-blindness, the inability to see in 3D using stereopsis, often sit in the 5-10% range. At the Curtin University HIVE visualization facility, we regularly show stereoscopic content. During those demonstrations we invariably show a test random dot stereogram and it’s been our casual observation that the incidence of stereo-blindness amongst visits has been much lower than the 5-10% figure, perhaps as low as 2%. Our thought was that perhaps eye care has improved since the time that the original stereo-blindness studies were performed. A VR based user study was recently run in the HIVE, as part of a PhD project, with an aim to study distance perception in underwater virtual heritage experiences. Distance perception is facilitated by a range of visual cues, including stereoscopic vision, and as a result we screened participants for stereo-blindness. Using a standardized stereo test, we found approximately 5% of participants reported as stereo blind. This presentation will provide some background on stereo-blindness, and discuss issues related to measuring stereo-blindness and its likely prevalence in the general population. It will also provide an early look at the VR based user study investigating distance perception in underwater virtual heritage experiences.

Dark stereo

It is often desirable or unavoidable to display Virtual Reality (VR) or stereoscopic content at low brightness. For example, a dimmer display reduces the flicker artefacts that are introduced by low-persistence VR headsets. It also saves power, prolongs battery life, and reduces the cost of a display or projection system. Additionally, stereo movies are usually displayed at relatively low luminance due to polarization filters or other optical elements necessary to separate two views. However, the binocular depth cues become less reliable at low luminance. In this paper, we propose a model of stereo constancy that predicts the precision of binocular depth cues for a given contrast and luminance. We use the model to design a novel contrast enhancement algorithm that compensates for the deteriorated depth perception to deliver good-quality stereoscopic images even for displays of very low brightness.

No-reference stereoscopic image quality assessment on both complex contourlet and spatial domain via Kernel ELM

Stereoscopic imaging is widely used in many fields. To guarantee the best quality of experience, it is necessary to design a robust and accurate quality assessment model for stereoscopic content. In this paper, we proposed a no-reference stereoscopic image quality assessment (NR-SIQA) model using both complex contourlet and spatial domain features of monocular and binocular images. Monocular features extracted from the CIELAB color space are exploited to characterize quality degradation, including the across-scale and across-orientation correlation in complex contourlet domain and natural scene statistics-based (NSS) features in spatial domain. Then, the binocular features consist of energy along with energy difference and structural correlation in complex contourlet domain, and statistics distribution in spatial domain extracted from the synthesized cyclopean image and the 3D visual discomfort measure based on the statistics of disparity map. Finally, the above features are mapped into the predicted quality scores by the Kernel ELM (KELM) regression model. Experimental results on four public datasets show that the proposed model is highly consistent with human subjective perception in terms of accuracy and generalization.

Neural network-based assessment of the impact induced in video quality assessment by the semantic labels

Subjective video quality assessment generally comes across with semantically labeled evaluation scales (e.g. Excellent, Good, Fair, Poor and Bad on a single stimulus, 5 level grading scale). While suspicions about an eventual bias these labels induce in the quality evaluation always occur, to the best of our knowledge, very few state-of-the-art studies target an objective assessment of such an impact. Our study presents a neural network solution in this respect. We designed a 5-class classifier, with 2 hidden layers, and a softmax output layer. An ADAM optimizer coupled to a Sparse Categorical Cross Entropy function is subsequently considered. The experimental results are obtained out of processing a database composed of 440 observers scoring about 7 hours of video content of 4 types (high-quality stereoscopic video content, low-quality stereoscopic video content, high-quality 2D video, and low-quality 2D video). The experimental results are discussed and confrontment to the reference given by a probability-based estimation method. They show an overall good convergence between the two types of methods while pointing out to some inner applicative differences that are discussed and explained.

Binocular Rivalry Oriented Predictive Autoencoding Network for Blind Stereoscopic Image Quality Measurement

Stereoscopic image quality measurement (SIQM) has become increasingly important for guiding stereo image processing and commutation systems due to the widespread usage of 3-D contents. Compared with conventional methods that are relied on handcrafted features, deep-learning-oriented measurements have achieved remarkable performance in recent years. However, most existing deep SIQM evaluators are not specifically built for stereoscopic contents and consider little prior domain knowledge of the 3-D human visual system (HVS) in network design. In this article, we develop a Predictive Auto-encoDing Network (PAD-Net) for blind/no-reference SIQM. In the first stage, inspired by the predictive coding theory that the cognition system tries to match bottom–up visual signal with top–down predictions, we adopt the encoder–decoder architecture to reconstruct the distorted inputs. Besides, motivated by the binocular rivalry phenomenon, we leverage the likelihood and prior maps generated from the predictive coding process in the Siamese framework for assisting SIQM. In the second stage, a quality regression network is applied to the fusion image for acquiring the perceptual quality prediction. The performance of PAD-Net has been extensively evaluated on three benchmark databases and the superiority has been well validated on both symmetrically and asymmetrically distorted stereoscopic images under various distortion types.

DP3 signal as a neuro-indictor for attentional processing of stereoscopic contents in varied depths within the ‘comfort zone’

When viewing stereoscopic 3D (S3D) displays, excessive binocular disparity is one of the major sources for the induced visual fatigue. Limiting the disparity within a certain range termed ‘comfort zone’ is a useful approach to alleviate the negative impact of viewing the S3D displays. Using the event-related potential (ERP) approach, we have previously showed that the differential P3 (DP3) signal evoked in an oddball task could be a useful neural indicator of the involvement of high-level cognitive functions in viewing S3D contents within the comfort zone compared to viewing 2D contents, despite of the similar subjective comfort of viewing experience for the two display conditions. It remains unclear whether the DP3 signal is sensitive enough for examination of the high-level processing when viewing S3D contents in different disparities within the comfort zone. Here, the DP3 signals were recorded while subjects performed an oddball task presented in S3D modality for three disparities (0.15, 0.45 and 0.76 arcdeg) within the comfort zone. Whereas the oddball task was of similar comfort by subjective measurement for different disparities, the DP3 signal showed monotonic delay in latency for increasing disparity, suggesting the capability of the DP3 signal for establishing a ‘cognitive comfort zone’ in terms of the high-level cognitive functions.

AdaBoost neural network and cyclopean view for no-reference stereoscopic image quality assessment

Stereoscopic imaging is widely used in many fields. In many scenarios, stereo images quality could be affected by various degradations, such as asymmetric distortion. Accordingly, to guarantee the best quality of experience, robust and accurate reference-less metrics are required for quality assessment of stereoscopic content. Most existing stereo no-reference Image Quality Assessment (IQA) models are not consistent with asymmetrical distortions. This paper presents a new no-reference stereoscopic image quality assessment metric using a human visual system (HVS) modeling and an advanced machine-learning algorithm. The proposed approach consists of two stages. In the first stage, cyclopean image is constructed considering the presence of binocular rivalry in order to cover the asymmetrically distorted part. In the second stage, gradient magnitude, relative gradient magnitude, and gradient orientation are extracted. These are used as a predictive source of information for the quality. In order to obtain the best overall performance against different databases, Adaptive Boosting (AdaBoost) idea of machine learning combined with artificial neural network model has been adopted. The benchmark LIVE 3D phase-I, phase-II, and IRCCyN/IVC 3D databases have been used to evaluate the performance of the proposed approach. Experimental results have demonstrated that the proposed metric performance achieves high consistency with subjective assessment and outperforms the blind stereo IQA over various types of distortion.

Visual Fatigue Assessment Based on Multi-task Learning

Abstract In recent years, with the rapid development of stereoscopic display technology, its applications have become increasingly popular in many fields, and, meanwhile, the number of audiences is also growing. The problem of visual fatigue is becoming more and more prominent. Visual fatigue is mainly caused by vergence‐accommodation conflicts. An evaluation experiment was conducted, and the electroencephalogram (EEG) data of the subjects were collected when they were watching stereoscopic content, and then the stereoscopic fatigue state of the subjects during the viewing process was analyzed. As deep learning is proved to be an effective end-to-end learning method and multi-task learning can alleviate the problem of lacking annotated data, the authors establish a user visual fatigue assessment model based on EEG by using multi-task learning, which can effectively obtain the user’s visual fatigue status, so as to make the comfort designs to avoid the harm caused by user’s visual fatigue.

A cognitive network management system to improve QoE in stereoscopic IPTV service

SummaryNew Internet Protocol Television (IPTV) services are including new technologies such as Stereoscopic TV and three‐dimensions (3D) HDTV. As well, increased ubiquitous networking and promoting in smart devices have led to high demand IPTV (over networks). Stereoscopic content required higher data flow to support these emerging TV services, and there are higher requirements at the network layer to provide good quality of service and quality of experience to the end users in delivering stereoscopic IPTV. In this paper, we propose a new concept of cognitive network management algorithm and protocol based on 3D coding techniques for delivering of stereoscopic IPTV service. The proposed approach explains how the management algorithm observes the network performance to guarantee the quality of the stereoscopic IPTV services, by measuring the performance of quality of service (QoS) parameters (delay, jitter, and packets loss) and quality of experience (QoE) metrics such as Peak Signal‐to‐Noise Ratio (PSNR), Moving Image Videography (MIV), and Mean Opinion Score (MOS). Those parameters are monitored in order to take appropriate codification decision for IPTV service provider. Moreover, the codification decision uses K‐mean classification to select the better codification for the end users. Therefore, both kinds of 3D coding formats such as Stereo Video Coding (SVC) format and 2D + Z Coding format (2D‐plus‐Depth) are selected in our experiments. As a result, our proposal successfully ensures the appropriate quality of service and quality of experience to the end users when the service of stereoscopic IPTV is being delivered.

Semantic label bias in subjective video quality evaluation: a standardization perspective

Recent studies brought to light that the semantic labels (e.g. Excellent, Good, Fair, Poor, and Bad) commonly associated with discrete scale ITU subjective quality evaluation induce a bias in MOS computation and that such a bias can be quantified by some reference coefficients which are independent with respect to the observers panel. The present paper reconsiders these results from a standard upgrading perspective. First, it theoretically investigates the way in which results obtained on semantically labeled scales can be “cleaned” from such an influence and derives the underlying computation formula for the mean opinion score. Secondly, it suggests a unitary evaluation procedure featuring both semantic free MOS computation and backward compatibility with respect to state-of-the-art solutions. The theoretical and methodological results are supported by subjective experiments corresponding to a total of 440 human observers, alternatively scoring 2D and stereoscopic video content. For each type of content, both high and low quality excerpts are alternatively considered. For each type of content and for each type of quality a 5 level (Excellent, Good, Fair, Poor, and Bad) grading scales is considered.

Gradient‐Guided Local Disparity Editing

AbstractStereoscopic 3D technology gives visual content creators a new dimension of design when creating images and movies. While useful for conveying emotion, laying emphasis on certain parts of the scene, or guiding the viewer's attention, editing stereo content is a challenging task. Not respecting comfort zones or adding incorrect depth cues, for example depth inversion, leads to a poor viewing experience. In this paper, we present a solution for editing stereoscopic content that allows an artist to impose disparity constraints and removes resulting depth conflicts using an optimization scheme. Using our approach, an artist only needs to focus on important high‐level indications that are automatically made consistent with the entire scene while avoiding contradictory depth cues and respecting viewer comfort.

Optimized GPU-based post-processing for stereoscopic multi-projector display systems

To drive multi-projector display systems, the image content has to be post-processed to apply geometrical and photometrical correction algorithms. Since the optical path is shared for both views of a stereoscopic projector, we propose to eliminate redundant calculations by processing both views at once. We show that by exploiting the color similarities of stereoscopic image pairs, the cache efficiency of LUT-based color correction methods can be improved. Stereoscopic content is often transmitted in different formats where a single frame represents both views in a spatially multiplexed pattern, and display devices typically support only a subset of these formats, so that the need for conversion arises. We propose an algorithm to directly incorporate such format conversions into the post-processing and show that our combined approach can achieve significant performance improvements compared to the naive multi-pass implementation. The reduced overhead of the post-processing step increases the number of projectors which can be driven by a single GPU, which enables building larger or higher resolution displays at lower costs. The flexibility gained by the ability to process different input formats also greatly enhances the usability of powerwalls by allowing to implement the post-processing at different levels and working with unmodified software applications.

Semi-Automatic 2D-to-3D Conversion Using Low-Rank Matrix Recovery

Semi-automatic 2D-to-3D conversion is a promising solution to 3D stereoscopic content creation. However, the depth continuous transition between user marked neighboring regions will be lost when user scribbles are sparse. To help solve this problem, a piecewise-continuity regularized low-rank matrix recovery method is developed. Our approach is based on the fact that a depth-map can be decomposed into a low-rank matrix and an outlier term matrix. First, an initial dense depth-map is interpolated from the user scribbles using matting Laplacian scheme under the assumption that depth-map is piecewise-continuous. Second, a piecewise-continuity constrained low-rank recovery model is developed to remove outliers which are introduced by the interpolation. Experimental comparisons with existing algorithms show that our method demonstrates significant advantage over depth continuous transition between neighboring regions.

Quality of Aesthetic Experience and Implicit Modulating Factors

Quality of Aesthetic Experience and Implicit Modulating Factors

Consumer-oriented Head Mounted Displays: Analysis and Evaluation of Stereoscopic Characteristics and User Preferences

The increasing availability of advanced but affordable technologies is providing interesting opportunities for the development of ICT applications for the social good. In particolar, the upcoming diffusion of advanced Head Mounted Displays dedicated to the consumer market has lead to a great interest in the design and production of dedicated media, like e.g. immersive movies and Virtual Reality applications. These devices use stereoscopic visualization to enhance the sense of realism and immersivity in a virtual scene. However, a correct stereoscopic visualization requires an accurate consideration of different parameters related to the production and display stages. In this paper, we analyze the stereoscopic setup of a Head Mounted Display, in order to highlight its main visualization characteristics in relation with the known issues and requirements for a correct stereoscopic visualization, and to establish some preliminary guidelines for an optimal creation of stereoscopic contents.

Stereoscopic Video Quality Assessment Using Binocular Energy

Stereoscopic imaging is becoming increasingly popular. However, to ensure the best quality of experience, there is a need to develop more robust and accurate objective metrics for stereoscopic content quality assessment. Existing stereoscopic image and video metrics are either extensions of conventional 2-D metrics (with added depth or disparity information) or are based on relatively simple perceptual models. Consequently, they tend to lack the accuracy and robustness required for stereoscopic content quality assessment. This paper introduces full-reference stereoscopic image and video quality metrics based on a human visual system (HVS) model incorporating important physiological findings on binocular vision. The proposed approach is based on the following three contributions. First, it introduces a novel HVS model extending previous models to include the phenomena of binocular suppression and recurrent excitation. Second, an image quality metric based on the novel HVS model is proposed. Finally, an optimized temporal pooling strategy is introduced to extend the metric to the video domain. Both image and video quality metrics are obtained via a training procedure to establish a relationship between subjective scores and objective measures of the HVS model. The metrics are evaluated using publicly available stereoscopic image/video databases as well as a new stereoscopic video database. An extensive experimental evaluation demonstrates the robustness of the proposed quality metrics. This indicates a considerable improvement with respect to the state-of-the-art with average correlations with subjective scores of 0.86 for the proposed stereoscopic image metric and 0.89 and 0.91 for the proposed stereoscopic video metrics.

Subjective and Objective Study of the Relation Between 3D and 2D Views Based on Depth and Bitrate

The tremendous growth in 3D (stereo) imaging and display technologies has led to stereoscopic content (video and image) becoming increasingly popular. However, both the subjective and the objective evaluation of stereoscopic video content has not kept pace with the rapid growth of the content. Further, the availability of standard stereoscopic video databases is also quite limited. In this work, we attempt to alleviate these shortcomings. We present a stereoscopic video database and its subjective evaluation. We have created a database containing a set of 144 distorted videos. We limit our attention to H.264 compression artifacts. The distorted videos were generated using 6 uncompressed pristine videos of left and right views originally created by Ecole Polytechnique Federal De Lausanne (EPFL)[1]. The reference video sequences contain a good combination of texture, motion, depth information and we divided these videos into 2 groups based on depth information. Further, 19 subjects participated in the subjective assessment task. Based on the subjective study, we have formulated a conditional relation between the 2D and stereoscopic subjective scores as a function of compression rate and depth range. We have also evaluated the performance of popular 2D and 3D image/video quality assessment (I/VQA) algorithms on our database.