Stereoscopic Image Pairs Research Articles

MCL-3D: A Database for Stereoscopic Image Quality Assessment using 2D-Image-Plus-Depth Source

A new stereoscopic image quality assessment database rendered using the 2D-image- plus-depth source, called MCL-3D, is described and the performance benchmarking of several known 2D and 3D image quality metrics using the MCL-3D database is presented in this work. Nine image-plus-depth sources are first selected, and a depth image- based rendering (DIBR) technique is used to render stereoscopic image pairs. Distortions applied to either the texture image or the depth image before stereoscopic image rendering include: Gaussian blur, additive white noise, down-sampling blur, JPEG and JPEG-2000 (JP2K) compression and transmission error. Furthermore, the distortion caused by imperfect rendering is also examined. The MCL-3D database contains 693 stereoscopic image pairs, where one third of them are of resolution 1024*768 and two thirds are of resolution 1920*1080. The pair-wise comparison was adopted in the subjective test for user friendliness, and the Mean Opinion Score (MOS) were computed accordingly. Finally, we evaluate the performance of several 2D and 3D image quality metrics applied to MCL-3D. All texture images, depth images, rendered image pairs in MCL- 3D and their MOS values obtained in the subjective test are available to the public (http: //mcl.usc.edu/mcl-3d-database/) for future research and development.

Semiautomatic Object-Space Road Extraction Combining a Stereoscopic Image Pair and a TIN-Based DTM

In this letter, a semiautomatic method for road extraction in object space is proposed that combines a stereoscopic pair of low-resolution aerial images with a digital terrain model (DTM) structured as a triangulated irregular network (TIN). First, we formulate an objective function in the object space to allow the modeling of roads in 3-D. In this model, the TIN-based DTM allows the search for the optimal polyline to be restricted along a narrow band that is overlaid upon it. Finally, the optimal polyline for each road is obtained by optimizing the objective function using the dynamic programming optimization algorithm. A few seed points need to be supplied by an operator. To evaluate the performance of the proposed method, a set of experiments was designed using two stereoscopic pairs of low-resolution aerial images and a TIN-based DTM with an average resolution of 1 m. The experimental results showed that the proposed method worked properly, even when faced with anomalies along roads, such as obstructions caused by shadows and trees.

Three-dimensional visual comfort assessment via preference learning

Three-dimensional (3-D) visual comfort assessment (VCA) is a particularly important and challenging topic, which involves automatically predicting the degree of visual comfort in line with human subjective judgment. State-of-the-art VCA models typically focus on minimizing the distance between predicted visual comfort scores and subjective mean opinion scores (MOSs) by training a regression model. However, obtaining precise MOSs is often expensive and time-consuming, which greatly constrains the extension of existing MOS-aware VCA models. This study is inspired by the fact that humans tend to conduct a preference judgment between two stereoscopic images in terms of visual comfort. We propose to train a robust VCA model on a set of preference labels instead of MOSs. The preference label, representing the relative visual comfort of preference stereoscopic image pairs (PSIPs), is generally precise and can be obtained at much lower cost compared with MOS. More specifically, some representative stereoscopic images are first selected to generate the PSIP training set. Then, we use a support vector machine to learn a preference classification model by taking a differential feature vector and the corresponding preference label of each PSIP as input. Finally, given a testing sample, by considering a full-round paired comparison with all the selected representative stereoscopic images, the visual comfort score can be estimated via a simple linear mapping strategy. Experimental results on our newly built 3-D image database demonstrate that the proposed method can achieve a better performance compared with the models trained on MOSs.

Hallucinating Stereoscopy from a Single Image

AbstractWe introduce a novel method for enabling stereoscopic viewing of a scene from a single pre‐segmented image. Rather than attempting full 3D reconstruction or accurate depth map recovery, we hallucinate a rough approximation of the scene's 3D model using a number of simple depth and occlusion cues and shape priors. We begin by depth‐sorting the segments, each of which is assumed to represent a separate object in the scene, resulting in a collection of depth layers. The shapes and textures of the partially occluded segments are then completed using symmetry and convexity priors. Next, each completed segment is converted to a union of generalized cylinders yielding a rough 3D model for each object. Finally, the object depths are refined using an iterative ground fitting process. The hallucinated 3D model of the scene may then be used to generate a stereoscopic image pair, or to produce images from novel viewpoints within a small neighborhood of the original view. Despite the simplicity of our approach, we show that it compares favorably with state‐of‐the‐art depth ordering methods. A user study was conducted showing that our method produces more convincing stereoscopic images than existing semi‐interactive and automatic single image depth recovery methods.

Oriented Correlation Models of Distorted Natural Images With Application to Natural Stereopair Quality Evaluation

In recent years, bandpass statistical models of natural, photographic images of the world have been used with great success to solve highly diverse problems involving image representation, image repair, image quality assessment (IQA), and image compression. One missing element has been a reliable and generic model of spatial image correlation that reflects the distributions of oriented and relatively oriented spatial structures. We have developed such a model for bandpass pristine images and have generalized it here to also capture the spatial correlation structure of bandpass distorted images. The model applies well to both luminance and depth images. As a demonstration of the usefulness of the generalized model, we develop a new no-reference stereoscopic/3D IQA framework, dubbed stereoscopic/3D blind image naturalness quality index, which utilizes both univariate and generalized bivariate natural scene statistics (NSS) models. We first validate the robustness and effectiveness of these novel bivariate and correlation NSS features extracted from distorted stereopairs, then demonstrate that they are predictive of distortion severity. Our experimental results show that the resulting 3D image quality predictor based in part on the new model outperforms state-of-the-art full- and no-reference 3D IQA algorithms on both symmetrically and asymmetrically distorted stereoscopic image pairs.

A depth perception and visual comfort guided computational model for stereoscopic 3D visual saliency

With the emerging development of three-dimensional (3D) related technologies, 3D visual saliency modeling is becoming particularly important and challenging. This paper presents a new depth perception and visual comfort guided saliency computational model for stereoscopic 3D images. The prominent advantage of the proposed model is that we incorporate the influence of depth perception and visual comfort on 3D visual saliency computation. The proposed saliency model is composed of three components: 2D image saliency, depth saliency and visual comfort based saliency. In the model, color saliency, texture saliency and spatial compactness are computed respectively and fused to derive 2D image saliency. Global disparity contrast is considered to compute depth saliency. Particularly, we train a visual comfort prediction function to distinguish stereoscopic image pair as high comfortable stereo viewing (HCSV) or low comfortable stereo viewing (LCSV), and devise different computational rules to generate a visual comfort based saliency map. The final 3D saliency map is obtained by using a linear combination and enhanced by a “saliency-center bias” model. Experimental results show that the proposed 3D saliency model outperforms the state-of-the-art models on predicting human eye fixations and visual comfort assessment.

Objective Evaluation Criteria for Shooting Quality of Stereo Cameras over Short Distance

Stereo cameras are the basic tools used to ob- tain stereoscopic image pairs, which can lead to truly great image quality. However, some inappropriate shooting condi- tions may cause discomfort while viewing stereo images. It is therefore considerably necessary to establish the percep- tual criteria that can be used to evaluate the shooting qual- ity of stereo cameras. This article proposes objective quality evaluation criteria based on the characteristics of parallel and toed-in camera configurations. Considering the differ- ent internal structures and basic shooting principles, this pa- per focuses on short-distance shooting conditions and estab- lishes assessment criteria for both parallel and toed-in cam- era configurations. Experimental results show that the pro- posed evaluation criteria can predict the visual perception of stereoscopic images and effectively evaluate stereoscopic image quality.

디지털 홀로그램으로부터 임의시점/임의시차 스테레오스코픽 영상 생성

This paper proposes a method to generate a stereoscopic image pair from a digital hologram by considering the situation that digital hologram data is serviced but the end-user does not have the proper display equipment, etc. We use Fresnel transform as the method to convert a digital hologram into an image. Each image of the stereoscopic image pair uses a part of the given digital hologram and the sizes of the two partial digital holograms for the two images are chosen to be the same. Here, the size of the image is adjusted by the size of the partial digital hologram and the disparity between the pair images is adjusted by the distance between the centers of the two partial hologram. This paper also deals with how to adjust the size and the disparity of the images. In this paper the generated stereoscopic images are implemented as an anaglyphic display type to confirm the feeling of distance by wearing the red-blue glasses.

A practical guide to the use of consumer-level digital still cameras for precise stereogrammetric <I>in situ</I> assessments in aquatic environments

Scientists planning to use underwater stereoscopic image technologies are often faced with numerous problems during the methodological implementations: commercial equipment is too expensive; the setup or calibration is too complex; or the imaging processing (i.e. measuring objects in the stereo-images) is too complicated to be performed without a time-consuming phase of training and evaluation. The present paper addresses some of these problems and describes a workflow for stereoscopic measurements for marine biologists. It also provides instructions on how to assemble an underwater stereo-photographic system with two digital consumer cameras and gives step-by-step guidelines for setting up the hardware. The second part details a software procedure to correct stereo-image pairs for lens distortions, which is especially important when using cameras with non-calibrated optical units. The final part presents a guide to the process of measuring the lengths (or distances) of objects in stereoscopic image pairs. To reveal the applicability and the restrictions of the described systems and to test the effects of different types of camera (a compact camera and an SLR type), experiments were performed to determine the precision and accuracy of two generic stereo-imaging units: a diver-operated system based on two Olympus Mju 1030SW compact cameras and a cable-connected observatory system based on two Canon 1100D SLR cameras. In the simplest setup without any correction for lens distortion, the low-budget Olympus Mju 1030SW system achieved mean accuracy errors (percentage deviation of a measurement from the object’s real size) between 10.2 and –7.6% (overall mean value: –0.6%), depending on the size, orientation and distance of the measured object from the camera. With the single lens reflex (SLR) system, very similar values between 10.1% and –3.4% (overall mean value: –1.2%) were observed. Correction of the lens distortion significantly improved the mean accuracy errors of either system. Even more, system precision (spread of the accuracy) improved significantly in both systems. Neither the use of a wide-angle converter nor multiple reassembly of the system had a significant negative effect on the results. The study shows that underwater stereophotography, independent of the system, has a high potential for robust and non-destructive in situ sampling and can be used without prior specialist training.

Salient object detection and classification for stereoscopic images

Stereoscopic images have become more and more prevalent following the rapid advances in 3D capturing and display techniques. However, there has been little research on visual content analysis for stereoscopic images. In this paper, we address the challenging problem of object detection and classification for stereoscopic images. An iterative method that can mutually boost salient object detection and object classification is proposed for stereoscopic images. This method includes two steps. In the first step, a 3D saliency detection method, which includes the contrastive and occlusion cues contained in each stereoscopic image pair along with the discriminative features provided by the SVM classifier, is proposed to localize object of interest in the stereoscopic images. In the second step, the bag of word features of foreground and background is pooled by using the localization information, and then is applied to train the SVM classifier. Each of the two steps benefits from the gradual improvement result in the other, no matter in the training or the testing process. To evaluate the performance of our approach, a 6-object class dataset of stereoscopic images real objects viewed under general lighting conditions, poses and viewpoints is set up. Our experimental results on the dataset, for object localization and object classification, demonstrate the effectiveness of the method.

Efficient hybrid tree-based stereo matching with applications to postcapture image refocusing.

Estimating dense correspondence or depth information from a pair of stereoscopic images is a fundamental problem in computer vision, which finds a range of important applications. Despite intensive past research efforts in this topic, it still remains challenging to recover the depth information both reliably and efficiently, especially when the input images contain weakly textured regions or are captured under uncontrolled, real-life conditions. Striking a desired balance between computational efficiency and estimation quality, a hybrid minimum spanning tree-based stereo matching method is proposed in this paper. Our method performs efficient nonlocal cost aggregation at pixel-level and region-level, and then adaptively fuses the resulting costs together to leverage their respective strength in handling large textureless regions and fine depth discontinuities. Experiments on the standard Middlebury stereo benchmark show that the proposed stereo method outperforms all prior local and nonlocal aggregation-based methods, achieving particularly noticeable improvements for low texture regions. To further demonstrate the effectiveness of the proposed stereo method, also motivated by the increasing desire to generate expressive depth-induced photo effects, this paper is tasked next to address the emerging application of interactive depth-of-field rendering given a real-world stereo image pair. To this end, we propose an accurate thin-lens model for synthetic depth-of-field rendering, which considers the user-stroke placement and camera-specific parameters and performs the pixel-adapted Gaussian blurring in a principled way. Taking ~1.5 s to process a pair of 640×360 images in the off-line step, our system named Scribble2focus allows users to interactively select in-focus regions by simple strokes using the touch screen and returns the synthetically refocused images instantly to the user.

Object-Coherence Warping for Stereoscopic Image Retargeting

This paper addresses the topic of content-aware stereoscopic image retargeting. The key to this topic is consistently adapting a stereoscopic image to fit displays with various aspect ratios and sizes while preserving visually salient content. Most methods focus on preserving the disparities and shapes of visually salient objects through nonlinear image warping, in which distortions caused by warping are propagated to homogenous and low-significance regions. However, disregarding the consistency of object deformation sometimes results in apparent distortions in both the disparities and shapes of objects. An object-coherence warping scheme is proposed to reduce this unwanted distortion. The basic idea is to utilize the information of matched objects rather than that of matched pixels in warping. Such information implies object correspondences in a stereoscopic image pair, which allows the generation of an object significance map and the consistent preservation of objects. This strategy enables our method to consistently preserve both the disparities and shapes of visually salient objects, leading to good content-aware retargeting. In the experiments, qualitative and quantitative analyses of various stereoscopic images show that our results are better than those generated by related methods in terms of consistency of object preservation.

Satellite stereoscopic pair images of very high resolution: a step forward for the development of landslide inventories

Development of landslide inventories based on remote sensing techniques has become one of the main tools in assessment of hazards and risk. Among those techniques, visual and automatic and semi-automatic analysis of high- and very high-resolution (VHR) satellite images, or a combination of these, has recently been considered as a promising way to identify and map landslides at local and regional scales. In this context, a landslide inventory for the municipality of Pahuatlan, Puebla, in central Mexico was prepared by combining three techniques: (1) visual analysis of stereoscopic pairs of VHR satellite images (GeoEye-1), (2) visual analysis of monoscopic VHR satellite images (SPOT 5 and Google Earth images), and (3) field surveying. In this paper, particular attention is given to landslide identification and mapping based on the GeoEye-1 stereo-pairs. Additionally, as a preliminary step in the use of VHR imagery, a general review is presented of the available VHR satellite images, software and hardware that can be useful for digital mapping of landslides. The landslide inventory included a total of 577 landslides, corresponding to an average density of 10.5 landslides per km2. Of these, 385 were classified as recent, 171 as old, and 21 as very old, regardless of state of activity. The total mapped area was 54.9 km2; 57.7 % of it had been affected by landsliding. The mean area occupied by recent landslides was of the order of 1,066 m2; for old landslides, it was 82,559 m2 and for very old landslides 1,173,952 m2. Debris flows were the most frequent type of movement (217), followed by 167 translational slides, 97 complex movements, 79 rotational slides, and 17 falls and topples. The cost–benefit relationships of a number of these techniques remain debatable because of the high cost of some of the VHR images and the related software and hardware. However, the appearance of new satellite sensors is likely to generate market competence, so this type of image will probably be available at a much lower cost in the near future. Additionally, it is important to consider that the use of several stereo-high-resolution images involves no cost, as downloading high-resolution images from Google Earth, using Google Earth Pro is currently available. The relative rapidity of these techniques can be highly valuable after a regional landslide disaster has occurred, since damage to roads and infrastructure usually prevents the rapid and accurate evaluation of the impact of landsliding. Most importantly, these techniques can be of great value for hazard evaluation of potentially unstable inhabited slopes.

Improved Tile Format of Stereoscopic Video for 3-D TV Broadcasting

Tile format is a new frame packing arrangement for the first generation of 3-D TV broadcasting services. It puts the left- and right-eye images of each stereoscopic image pair into a single composition frame that allows the service provider to reuse existing production and distribution infrastructure for offering 3-D TV services. Different from frame compatible formats, the tile format composition does not down-sample the left- and right-eye images and, thus, provides a higher video quality for both 3-D and 2-D display. The problem with the tile format is that it creates artificial edges in the right-eye image and introduces visible tiling artifacts around the artificial edges after compression. This brief paper presents a simple modification to the tile format composition method that significantly reduces the tiling artifacts. Experimental results show that local PSNRs around artificial edges are improved by over 1 dB.

Measurement of 3D Displacement Fields and Deformation Using 3D Laser Scanner and CCD Camera (0-10 mm)

Many applications dealing with the mechanical behavior of materials require the measurement of displacement fields or deformation fields. For this type of measurement, optical methods have become unavoidable due to their non-intrusive approach, their high spatial resolution, their high sensitivity, the large size of their examinated field, and the increasing power of the computers that now allow the processing of huge quantities of data. In this context we have developed a system for measuring 3-D displacement fields using CCD (camera charge digital) and 3D laser scanner .This method allows the measure of the 3-D displacement field using at least two pairs of stereoscopic images of an object corresponding to two states of its deformation (or the processing of a sequence of pairs of images acquired during the deformation). The main topics developed in this study are: the calibration of a camera and a 3D laser scanner, and measuring 3D displacement fields from coupling of camera and 3D laser sensor. The potential application of our method is the characterization of structures, for example, to provide assistance in the development of tools for stamping.

Changing Perspective in Stereoscopic Images

Traditional image editing techniques cannot be directly used to edit stereoscopic ("3D") media, as extra constraints are needed to ensure consistent changes are made to both left and right images. Here, we consider manipulating perspective in stereoscopic pairs. A straightforward approach based on depth recovery is unsatisfactory: Instead, we use feature correspondences between stereoscopic image pairs. Given a new, user-specified perspective, we determine correspondence constraints under this perspective and optimize a 2D warp for each image that preserves straight lines and guarantees proper stereopsis relative to the new camera. Experiments verify that our method generates new stereoscopic views that correspond well to expected projections, for a wide range of specified perspective. Various advanced camera effects, such as dolly zoom and wide angle effects, can also be readily generated for stereoscopic image pairs using our method.

Content-Driven Retargeting of Stereoscopic Images

This letter proposes a novel warping-based method for the content-driven retargeting of stereoscopic images. Conventional algorithms in single image retargeting generally do not consider the scene depth saliency and disparity consistency when applied independently to left and right images . Therefore, the salient region and stereoscopic correlation of independently retargeted images can become corrupted. On the other hand, the proposed algorithm retains the stereo consistency of the retargeted images by matching the vertices of the grid and preserving the correspondence between them. Vertex disparity is propagated by a GPU interpolation to construct a sparse disparity map. To improve the capability of preserving visually important regions, the sparse disparity is used in conjunction with the image gradient in a saliency map computation. The experimental results show that the proposed method retains the correct disparity, while the salient objects remain undistorted in the retargeted stereoscopic image pairs.

Lightweight spatial-multiplexed dual focal-plane head-mounted display using two freeform prisms

Accommodation and convergence play critical roles in the natural viewing of three-dimensional (3D) scenes, and these must be accurately matched to avoid visual fatigue. However, conventional stereoscopic head-mounted displays lack the ability to adjust accommodation cues. This is because they only have a single, fixed image plane, but the 3D virtual objects generated by a pair of stereoscopic images are displayed at different depths, either in front or behind the focal plane. Therefore, in order to view objects clearly, the eyes are forced to converge on those objects while maintaining accommodation fixed on the image plane. By employing freeform optical surfaces, we design a lightweight and wearable spatial-multiplexed dual focal-plane head-mounted display. This display can adjust the accommodation cue in accordance with the convergence cue as well as generate the retinal blur cue. The system has great potential applications in both scientific research and commercial market.

Casual Stereoscopic Photo Authoring

Stereoscopic 3D displays become more and more popular these years. However, authoring high-quality stereoscopic 3D content remains challenging. In this paper, we present a method for easy stereoscopic photo authoring with a regular (monocular) camera. Our method takes two images or video frames using a monocular camera as input and transforms them into a stereoscopic image pair that provides a pleasant viewing experience. The key technique of our method is a perceptual-plausible image rectification algorithm that warps the input image pairs to meet the stereoscopic geometric constraint while avoiding noticeable visual distortion. Our method uses spatially-varying mesh-based image warps. Our warping method encodes a variety of constraints to best meet the stereoscopic geometric constraint and minimize visual distortion. Since each energy term is quadratic, our method eventually formulates the warping problem as a quadratic energy minimization which is solved efficiently using a sparse linear solver. Our method also allows both local and global adjustments of the disparities, an important property for adapting resulting stereoscopic images to different viewing conditions. Our experiments demonstrate that our spatially-varying warping technique can better support casual stereoscopic photo authoring than existing methods and our results and user study show that our method can effectively use casually-taken photos to create high-quality stereoscopic photos that deliver a pleasant 3D viewing experience.

Virtual Table-Teleporter: Image Processing and Rendering for Horizontal Stereoscopic Display

We describe a new architecture composed of software and hardware for displaying stereoscopic images over a horizontal surface. It works as a ``Virtual Table and Teleporter'', in the sense that virtual objects depicted over a table have the appearance of real objects. This system can be used for visualization and interaction. We propose two basic configurations: the Virtual Table, consisting of a single display surface, and the Virtual Teleporter, consisting of a pair of tables for image capture and display. The Virtual Table displays either 3D computer generated images or previously captured stereoscopic video and can be used for interactive applications. The Virtual Teleporter captures and transmits stereoscopic video from one table to the other and can be used for telepresence applications. In both configurations the images are properly deformed and displayed for horizontal 3D stereo. In the Virtual Teleporter two cameras are pointed to the first table, capturing a stereoscopic image pair. These images are shown on the second table that is, in fact, a stereoscopic display positioned horizontally. Many applications can benefit from this technology such as virtual reality, games, teleconferencing, and distance learning. We present some interactive applications that we developed using this architecture.