Stereoscopic Image Sequences Research Articles

3D object articulation and motion estimation in model-based stereoscopic videoconference image sequence analysis and coding

This paper describes a procedure for model-based analysis and coding of both left and right channels of a stereoscopic image sequence. The proposed scheme starts with a hierarchical dynamic programming technique for matching across the epipolar line for efficient disparity/depth estimation. Foreground/background segmentation is initially based on depth estimation and is improved using motion and luminance information. The model is initialised by the adaptation of a wireframe model to the consistent depth information. Robust classification techniques are then used to obtain an articulated description of the foreground of the scene (head, neck, shoulders). The object articulation procedure is based on a novel scheme for the segmentation of the rigid 3D motion fields of the triangle patches of the 3D model object. Spatial neighbourhood constraints are used to improve the reliability of the original triangle motion estimation. The motion estimation and motion field segmentation procedures are repeated iteratively until a satisfactory object articulation emerges. The rigid 3D motion is then re-computed for each sub-object and finally, a novel technique is used to estimate flexible motion of the nodes of the wireframe from the rigid 3D motion vectors computed for the wireframe triangles containing each specific node. The performance of the resulting analysis and compression method is evaluated experimentally.

Object-based coding of stereoscopic and 3D image sequences

The author begins by discussing what object based coding is and goes on to consider the structure of object based stereoscopic coders. Different techniques for object-based stereoscopic image sequence coding are reviewed. These techniques basically differ in the way they define models and estimate model parameters. We review the various models used for representing motion and structure. Then we review segmentation techniques, and discuss coding of object parameters and image synthesis.

Object-based coding of stereo image sequences using joint 3-D motion/disparity compensation

An object-based coding scheme is proposed for the coding of a stereoscopic image sequence using motion and disparity information. A hierarchical block-based motion estimation approach is used for initialization, while disparity estimation is performed using a pixel-based hierarchical dynamic programming algorithm. A split-and-merge segmentation procedure based on three-dimensional (3-D) motion modeling is then used to determine regions with similar motion parameters. The segmentation part of the algorithm is interleaved with the estimation part in order to optimize the coding performance of the procedure. Furthermore, a technique is examined for propagating the segmentation information with time. A 3-D motion-compensated prediction technique is used for both intensity and depth image sequence coding. Error images and depth maps are encoded using discrete cosine transform (DCT) and Huffman methods. Alternately, an efficient wireframe depth modeling technique may be used to convey depth information to the receiver. Motion and wireframe model parameters are then quantized and transmitted to the decoder along with the segmentation information. As a straightforward application, the use of the depth map information for the generation of intermediate views at the receiver is also discussed. The performance of the proposed compression methods is evaluated experimentally and is compared to other stereoscopic image sequence coding schemes.

The use of stereo and motion in a generic object-based coder

Abstract Current developments of object-based coders target small image formats, low bit-rate and videophone applications with head-and-shoulders scenes. An object-based coder which can be used for any type of scene and can provide better picture quality than MPEG-2 at the same bit-rate is proposed in this paper. It is a high-quality object-based coder which aims to introduce the concept of object-based coding in quality-picture applications. The coder uses motion and stereo information throughout for image analysis and synthesis, and object ordering. The coder can be extended to a stereoscopic coder with only minor modifications. Results using CCIR 601 stereoscopic image sequences shot within RACE DISTIMA are presented.

Three-dimensional camera motion estimation and foreground/background separation for stereoscopic image sequences

This work examines the problem of foreground/background extraction when stereoscopic image information is available. A two-stage algorithm is proposed, followed by postprocessing for the estimation of the global motion in the scene and the identification of moving objects. At the first stage, an estimate of the 2-D global motion in the scene is formed which then serves as an initial estimate for the algorithm for a robust 3-D global motion estimate. The 3-D global motion estimation procedure is interleaved with a procedure for removal of the outlier vec- tors from the initial set of training vectors. Postprocessing of the foreground/background map produced results in an improved motion de- tection mask, which may then be used for 3-D motion compensation. © 1997 Society of Photo-Optical Instrumentation Engineers.

Depth-based segmentation

The tool presented in this paper performs an automatic segmentation of stereoscopic image sequences, based on the modeling of distance maps obtained by image processing. Two three-dimensional (3-D) image analysis algorithms are combined: (i) estimation of dense depth maps from stereoscopic image sequences and (ii) depth-based object segmentation. A Markovian statistical approach for such a segmentation of a dense depth map into arbitrarily shaped and oriented planar surfaces is described in detail. The simulation results on sequences Fun fair and Tunnel, provided on video tape to the MPEG-4 tests of November 1995, are discussed.

Joint motion/disparity MAP estimation for stereo image sequences

An algorithm is described for the joint estimation of motion and disparity vector fields from stereoscopic image sequences. Markov random fields (MRF) are used to model local interaction processes. Interaction of neighbouring-motion disparity vectors across a discontinuity line is prohibited via hidden Markov fields signalling discontinuities in the vector fields. Occlusion processes are also used to mark occluded image locations, which may yield ambiguous matches. The coherence of motion and disparity vector fields is exploited by means of the epipolar constraint and the so called ‘loop constraint’. A computationally efficient, suboptimal, hierarchical, deterministic relaxation algorithm is proposed.

Stereo image sequence coding based on three-dimensional motion estimation and compensation

A method that performs 3-D motion estimation for stereoscopic image sequences is presented. The 2-D motion of each object observed in one of the two views is modelled using a 3-D motion model involving a translation and a rotation. The estimation of model parameters is performed in two steps: a linear step involving 2-D vectors that are initially estimated using block matching techniques followed by a non-linear step involving displaced frame difference minimization. The regions where the 3-D model is applied are identified using a motion-based split and merge technique. Furthermore, an extension of the 3-D motion estimation method that uses a single 3-D motion model to describe the apparent 2-D motion in both channels is examined. These 3-D motion estimation methods are then integrated in a stereoscopic interframe coding scheme. A hybrid coder using block-based coding as a fall-back mode in cases where 3-D motion estimation fails, is proposed. Experimental results demonstrate the performance of the proposed coding scheme.

Model-based 3-D scene analysis from stereoscopic image sequences

A vision-based 3-D scene analysis system is described that is capable to model complex real-world scences like buildings automatically from stereoscopic image pairs. Input to the system is a sequence of stereoscopic images taken with two standard CCD Cameras and TV lenses. The relative orientation of both cameras to each other is known by calibration. The camera pair is then moved throughout the scene and a long sequence of closely spaced views is recorded. Each of the stereoscopic image pairs is rectified and a dense map of 3-D surface points is obtained by area correlation, object segmentation, interpolation, and triangulation. 3-D camera motion relative to the scene coordinate system is tracked directly from the image sequence which allows to fuse 3-D surface measurements from different view points into a consistent 3-D model scence. The surface geometry of each scene object is approximated by a triangular surface mesh which stores the surface texture in a texture map. From the textured 3-D models, realistic looking image sequences from arbitrary view points can be synthesized using computer graphics.

Evaluation of multiresolution block matching techniques for motion and disparity estimation

Multiresolution block matching methods for both monocular and stereoscopic image sequence coding are evaluated. These methods are seen to drastically reduce the amount of processing needed for block correspondence without seriously affecting the quality of the reconstructed images. The evaluation criteria are the prediction error and the speed of the algorithm for motion, disparity, and fused motion and disparity estimation, in comparison with the full search (exhaustive) method. A new method is also proposed based in multiresolution techniques, for efficient coding of the disparity or the displacement vector field.

Automatic Reconstruction of Buildings from Stereoscopic Image Sequences

A vision–based 3‐D scene analysis system is described that is capable to model complex real–world scenes like streets and buildings automatically from stereoscopic image pairs. Input to the system is a sequence of stereoscopic images taken with two standard CCD Cameras and TV lenses. The relative orientation of both cameras to each other is known by calibration. The camerapair is then moved throughout the scene and a long sequence of closely spaced views is recorded. Each of the stereoscopic image pairs is rectified and a dense map of 3‐D suface points is obtained by area correlation, object segmentation, interpolation, and triangulation. 3‐D camera motion relative to the scene coordinate system is tracked directly from the image sequence which allows to fuse 3‐D surface measurements from different viewpoints into a consistent 3‐D model scene. The surface geometry of each scene object is approximated by a triangular surface mesh which stores the suface texture in a texture map. From the textured 3‐D models, realistic looking image sequences from arbitrary view points can be synthesized using computer graphics.

Constrained disparity and motion estimators for 3DTV image sequence coding

This paper presents two-dimensional motion estimation methods which take advantage of the intrinsic redundancies inside 3DTV stereoscopic image sequences. Most of the previous studies extract, either disparity vector fields if they are involved in stereovision, or apparent motion vector fields to be applied to motion compensation coding schemes. For 3DTV image sequence analysis and transmission, we can jointly estimate these two feature fields. Locally, initial image data are grouped within two views (the left and right ones) at two successive time samples and spatio-temporal coherence has to be used to enhance motion vector field estimation. Three different levels of ‘coherence’ have been experimented subject to the crucial assumption that an initial calibration of the stereoscopic sensors provides us with geometric change of coordinates for two matched features.