Multi-view Image Sequences Research Articles

A SYSTEMATIC COMPARISON OF DIRECT AND IMAGE-BASED GEOREFERENCING IN CHALLENGING URBAN AREAS

Image-based mobile mapping systems enable an efficient acquisition of georeferenced image sequences, which can be used for geo-data capture in subsequent steps. In order to provide accurate measurements in a given reference frame while e.g. aiming at high fidelity 3D urban models, high quality georeferencing of the captured multi-view image sequences is required. Moreover, sub-pixel accurate orientations of these highly redundant image sequences are needed in order to optimally perform steps like dense multi-image matching as a prerequisite for 3D point cloud and mesh generation. While direct georeferencing of image-based mobile mapping data performs well in open areas, poor GNSS coverage in urban canyons aggravates fulfilling these high accuracy requirements, even with high-grade inertial navigation equipment. Hence, we conducted comprehensive investigations aiming at assessing the quality of directly georeferenced sensor orientations as well as the expected improvement by image-based georeferencing in a challenging urban environment. Our study repeatedly delivered mean trajectory deviations of up to 80 cm. By performing image-based georeferencing using bundle adjustment for a limited set of cameras and a limited number of ground control points, mean check point residuals could be lowered from approx. 40 cm to 4 cm. Furthermore, we showed that largely automated image-based georeferencing is capable of detecting and compensating discontinuities in directly georeferenced trajectories.

Multiview video quality enhancement

Realistic visualization is crucial for a more intuitive representation of complex data, medical imaging, simulation, and entertainment systems. In this respect, multiview autostereoscopic displays are a great step toward achieving the complete immersive user experience, although providing high-quality content for these types of displays is still a great challenge. Due to the different characteristics/settings of the cameras in the multiview setup and varying photometric characteristics of the objects in the scene, the same object may have a different appearance in the sequences acquired by the different cameras. Images representing views recorded using different cameras, in practice, have different local noise, color, and sharpness characteristics. View synthesis algorithms introduce artifacts due to errors in disparity estimation/bad occlusion handling or due to an erroneous warping function estimation. If the input multiview images are not of sufficient quality and have mismatching color and sharpness characteristics, these artifacts may become even more disturbing. Accordingly, the main goal of our method is to simultaneously perform multiview image sequence denoising, color correction, and the improvement of sharpness in slightly defocused regions. Results show that the proposed method significantly reduces the amount of the artifacts in multiview video sequences, resulting in a better visual experience.

An Improved Hidden Markov Model Based on Human Motion Detecting Approach via Multi-View Image Sequences

An Improved Hidden Markov Model Based on Human Motion Detecting Approach via Multi-View Image Sequences

Multi-View Human Action Recognition Using Wavelet Data Reduction and Multi-Class Classification

Human action recognition from video has several potential to apply in different real-life applications, but the most cases in this field suffer from the variation in viewpoint. Most of published methods in this area are considered the performance of each single camera, therefore the change in the viewpoints significantly decrease the recognition rate. In this paper, multiple views are considered together and a method has proposed to recognize human action depicted in multi-view image sequences. In the first step, the border of the human body's silhouette is extracted and distance signal is calculated. In the next step, the wavelet transform is applied to extract coefficients of single-view features, and then the extracted features are combined to compose multi-view features. Finally a hierarchical classifier using support vector machine and Naive Bayes classifiers is implemented to classify the actions. The average of overall action recognition accuracy for 12 actions using 5 different angles of views on the IXMAS dataset is 88.22. The results of experiments on the popular multi-view dataset have shown the proposed method achieves high and state-of-the-art success rates. In other word, combination of single-view extracted features from the wavelet approximation coefficients and composing the multi-view features can be used as the multi-view features. Further, the hierarchical classifier can be applied to recognize actions in multi-view human action recognition area.

Capturing Human Motion based on Modified Hidden Markov Model in Multi-View Image Sequences

Human motion capturing is of great importance in video information retrieval, hence, in this paper, we propose a novel approach to effectively capturing human motions based on modified hidden markov model from multi-view image sequences. Firstly, the structure of the human skeleton model is illustrated, which is extended from skeleton root and spine root, and this skeleton consists of right leg, left leg and spine. Secondly, our proposed human motion capturing system is made up of data training module and human motion capturing module. In the data training module, multi-views motion information is extracted from a human motion database, and feature database of human motion capturing is constructed through combining multi-views motions. In the human motion capturing module, results of motion capturing can be achieved through motion classification based on a modified hidden markov model. Thirdly, the modified hidden markov model is designed by utilizing the fuzzy measure, fuzzy integer, and fuzzy intersection operator through a scaling process. Finally, a standard motion capture dataset- MPI08_Database is utilized to make performance evaluation. Compared with the existing methods, the proposed approach can effectively capture human motions with high precision

Interactive object segmentation from multi-view images

Despite the great progress on interactive image segmentation, image co-segmentation, 2D and 3D segmentation, there is still no workable solution to the problem: given a set of calibrated or un-calibrated multi-view images (say, more than 40 images), by interactively cutting 3∼4 images, can the foreground object of the rest images be quickly cutout automatically and accurately? In this paper, we propose a non-trivial engineering solution to this problem. Our basic idea is to integrate 3D segmentation with 2D segmentation so as to combine their advantages. Our proposed system iteratively performs 2D and 3D segmentation, where the 3D segmentation results are used to initialize 2D segmentation and ensure the silhouette consistency among different views and the 2D segmentation results are used to provide more accurate cues for the 3D segmentation. The experimental results show that the proposed system is able to generate highly accurate segmentation results, even for some challenging real-world multi-view image sequences, with a small amount of user input.

Compressed-sensing recovery of multiview image and video sequences using signal prediction

In the compressed sensing of multiview images and video sequences, signal prediction is incorporated into the reconstruction process in order to exploit the high degree of interview and temporal correlation common to multiview scenarios. Instead of recovering each individual frame independently, neighboring frames in both the view and temporal directions are used to calculate a prediction of a target frame, and the difference is used to drive a residual-based compressed-sensing reconstruction. The proposed approach demonstrates a significant gain in reconstruction quality relative to the straightforward compressed-sensing recovery of each frame independently of the others in the multiview set, as well as a significant performance advantage as compared to a pair of benchmark multiple-frame compressed-sensing reconstructions.

Optimization and Soft Constraints for Human Shape and Pose Estimation Based on a 3D Morphable Model

We propose an approach about multiview markerless motion capture based on a 3D morphable human model. This morphable model was learned from a database of registered 3D body scans in different shapes and poses. We implement pose variation of body shape by the defined underlying skeleton. At the initialization step, we adapt the 3D morphable model to the multi-view images by changing its shape and pose parameters. Then, for the tracking step, we implement a method of combining the local and global algorithm to do the pose estimation and surface tracking. And we add the human pose prior information as a soft constraint to the energy of a particle. When it meets an error after the local algorithm, we can fix the error using less particles and iterations. We demonstrate the improvements with estimating result from a multi-view image sequence.

Efficient disparity vector prediction schemes with modified P frame for 2D camera arrays

An efficient disparity estimation algorithm for multi-view video sequences, recorded by a two-dimensional camera array in which the cameras are spaced equidistantly, is presented. Because of the strong geometrical relationship among views, the disparity vectors of a certain view can for most blocks be derived from the disparity vectors of other views. A frame constructed using that idea is called a D frame in this work. Three new prediction schemes which contain D frames are proposed for encoding 5 × 3 multi-view video sequences. The schemes are applied to several multi-view image sequences taken from a camera-array and they are compared in terms of quality, bit-rate and complexity. The experimental results show that the proposed prediction schemes significantly decrease the complexity of the encoder at a very low cost of quality and/or bit-rate.

Image Recognition With a Limited Number of Pixels for Visual Prostheses Design

With the rapid development and crossover among the information science, microelectronics, material science, and biomedical disciplines, the visual prosthesis makes visual reparation possible. Because the number of stimulation electrodes is strictly limited by various complicated factors, it is necessary to determine the minimum visual requirements to achieve useful artificial vision for image recognition. This research has studied how many pixels individual images need to have to be correctly and economically recognized by blind subjects. In order to extract the figure of the image with a limited number of pixels, we have proposed a wavelet-based image processing methods, and six resolutions (8 × 8, 16 × 16, 24 × 24, 32 × 32, 48 × 48, and 64 × 64) are investigated. Psychophysical experiments have been designed to verify our proposed image processing method and to investigate the recognition accuracy with a limited number of pixels. The results show that the recognition accuracy increases with the number of pixels. The recognition accuracy varied with tested images, when a resolution of 24 × 24 was used: six of the eight image objects were recognized with an accuracy of >50%, and the remaining two of the eight image objects were recognized with an accuracy of <50%. Moreover, when the resolution is more than 32 × 32, the increase of the recognition accuracy is no longer obvious. We also have investigated the impact of different perspectives of the same object to the recognition accuracy. The experiment shows that providing multiview image sequences, subjects can receive more visual information to obtain higher recognition accuracy.

Reconstruction of photorealistic 3D model of ceramic artefacts for interactive virtual exhibition

Abstract To create photorealistic three-dimensional (3D) models of real scenes and objects is a challenging problem that demands advanced knowledge of computer vision and computer graphics. Systems that can reconstruct the 3D model of cultural artefacts have found many applications such as virtual museum and historical archiving. While there are methods for 3D digitization of cultural artefacts with high geometric resolution, there are still limitations in achieving high textural resolution for virtual exhibition. One major problem is that the object surface exhibits specular reflection of illuminated light during the acquisition of surface texture. The shading of the target object does not match to other objects or pictures in the virtual scene. Also, if texture of the object must be composed of multiple images, the mismatch of shading (radiometric difference) among the images can be very prominent. In order to create high visual quality exhibition, the specular reflections must be eliminated and then the virtual scene is relit by a synthetic light source. Most existing methods for the identification and removal of specular reflection component demand special device or rely on information obtained in a single image. In order to reconstruct a complete 3D model, we need to acquire a multi-view image sequence. We therefore propose a new method that is applicable for the separation of diffuse and specular reflection components in multi-view image sequence. Also, our method can tackle the specular reflection across the texture boundary. The image sequence is first normalized by the estimated illumination color. Based on the dichromatic reflection model, the specular chromaticity is replaced by the corresponding diffuse chromaticity, which can always be found in neighboring views with the highlights already faded away. We test the new method in modeling Yixing ceramic teapots. The shape model of the teapot is obtained by a laser scanner. The diffuse image sequence is then used to generate the texture map. We create the virtual scene with the photorealistic 3D teapot model, some synthetic 3D models and still pictures. Interactive exhibition of the artefact is achieved with the control of the mouse and simple keyboard commands. This paper gives an account of the procedures for the creation of interactive virtual exhibition of ceramic artefacts.

Human action recognition using shape and CLG-motion flow from multi-view image sequences

In this paper, we present a method for human action recognition from multi-view image sequences that uses the combined motion and shape flow information with variability consideration. A combined local–global (CLG) optic flow is used to extract motion flow feature and invariant moments with flow deviations are used to extract the global shape flow feature from the image sequences. In our approach, human action is represented as a set of multidimensional CLG optic flow and shape flow feature vectors in the spatial–temporal action boundary. Actions are modeled by using a set of multidimensional HMMs for multiple views using the combined features, which enforce robust view-invariant operation. We recognize different human actions in daily life successfully in the indoor and outdoor environment using the maximum likelihood estimation approach. The results suggest robustness of the proposed method with respect to multiple views action recognition, scale and phase variations, and invariant analysis of silhouettes.

An Epipolar Geometry-Based Fast Disparity Estimation Algorithm for Multiview Image and Video Coding

Effectively coding multiview visual content is an indispensable research topic because multiview image and video that provide greatly enhanced viewing experiences often contain huge amounts of data. Generally, conventional hybrid predictive-coding methodologies are adopted to address the compression by exploiting the temporal and interviewpoint redundancy existing in a multiview image or video sequences. However, their key yet time-consuming component, motion estimation (ME), is usually not efficient in interviewpoint prediction or disparity estimation (DE), because interviewpoint disparity is completely different from temporal motion existing in the conventional video. Targeting a generic fast DE framework for interviewpoint prediction, we propose a novel DE technique in this paper to accelerate the disparity search by employing epipolar geometry. Theoretical analysis, optimal disparity vector distribution histograms, and experimental results show that the proposed epipolar geometry-based DE can greatly reduce search region and effectively track large and irregular disparity, which is typical in convergent multiview camera setups. Compared with the existing state-of-the-art fast ME approaches, our proposed DE can obtain a similar coding efficiency while achieving a significant speedup for interviewpoint prediction and coding. Moreover, a robustness study shows that the proposed DE algorithm is insensitive to the epipolar geometry estimation noise. Hence, its wide application for multiview image and video coding is promising

Multiview Image Coding Based on Geometric Prediction

Many existing multiview image/video coding techniques remove inter-viewpoint redundancy by applying disparity compensation in a conventional video coding framework, e.g., H.264/MPEG-4 AVC. However, conventional methodology works ineffectively as it ignores the special characteristics of inter-viewpoint disparity. In this paper, we propose a geometric prediction methodology for accurate disparity vector (DV) prediction, such that we can largely reduce the disparity compensation cost. Based on the new DV predictor, we design a basic framework that can be implemented in most existing multiview image/video coding schemes. We also use state-of-the-art H.264/MPEG-4 AVC as an example to illustrate how the proposed framework can be integrated with conventional video coding algorithms. Our experiments show proposed scheme can effectively tracks disparity and greatly improves coding performance. Compared with H.264/MPEG-4 AVC codec, our scheme outperforms maximally 1.5 dB when encoding some typical multiview image sequences. We also carry out an experiment to evaluate the robustness of our algorithm. The results indicate our method is robust and can be used in practical applications.

On 3-D scene flow and structure recovery from multiview image sequences

Two novel systems computing dense three-dimensional (3-D) scene flow and structure from multiview image sequences are described in this paper. We do not assume rigidity of the scene motion, thus allowing for nonrigid motion in the scene. The first system, integrated model-based system (IMS), assumes that each small local image region is undergoing 3-D affine motion. Non-linear motion model fitting based on both optical flow constraints and stereo constraints is then carried out on each local region in order to simultaneously estimate 3-D motion correspondences and structure. The second system is based on extended gradient-based system (EGS), a natural extension of two-dimensional (2-D) optical flow computation. In this method, a new hierarchical rule-based stereo matching algorithm is first developed to estimate the initial disparity map. Different available constraints under a multiview camera setup are further investigated and utilized in the proposed motion estimation. We use image segmentation information to adopt and maintain the motion and depth discontinuities. Within the framework for EGS, we present two different formulations for 3-D scene flow and structure computation. One formulation assumes that initial disparity map is accurate, while the other does not. Experimental results on both synthetic and real imagery demonstrate the effectiveness of our 3-D motion and structure recovery schemes. Empirical comparison between IMS and EGS is also reported.

Unsupervised recognition of multi-view face sequences based on pairwise clustering with attraction and repulsion

In this paper we propose and investigate the possibilities inherent in a new, unsupervised approach to multi-view face recognition, which can be formulated mathematically as a problem of partitioning of proximity data, obtained from multi-view face image sequences. The proposed approach is implemented in two novel pairwise clustering algorithms, CAR1 and CAR2, which partition the input data into identity clusters by performing combinatorial optimization guided by two types of interaction forces, attraction and repulsion, imposed on the original proximity matrices. Several experiments were conducted in order to test the performance of the proposed algorithms on real-world datasets including both frontal and side-view faces, which have been gathered over a period of several months. The obtained results can be considered encouraging for the general approach proposed here, and the new algorithms compared favorably to two other pairwise clustering algorithms, recently proposed in the image segmentation literature.

Rigid and non-rigid 3D motion estimation from multiview image sequences

Multiview image sequence processing has been the focus of considerable attention in recent literature. This paper presents an efficient technique for object-based rigid and non-rigid 3D motion estimation, applicable to problems occurring in multiview image sequence coding applications. More specifically, a neural network is formed for the estimation of the rigid 3D motion of each object in the scene, using initially estimated 2D motion vectors corresponding to each camera view. Non-linear error minimization techniques are adopted for neural network weight update. Furthermore, a novel technique is also proposed for the estimation of the local non-rigid deformations, based on the multiview camera geometry. Experimental results using both stereoscopic and trinocular camera setups illustrate and evaluate the proposed scheme.

Layered representation of scenes based on multiview image analysis

This paper describes a novel cooperative procedure for the segmentation of multiview image sequences exploiting multiple sources of information. Compared to other approaches, no a priori information is needed about the structure and the arrangement of objects in the scene. Three cameras in a particular unsymmetrical set-up are used in the system. The color distribution and the object contours in the constituent 2-D images, the disparity information in stereo image pairs, as well as motion information in subsequent images, are analyzed and evaluated in a cooperative procedure to get reliable segmentation results. The scene is decomposed into a variable number of depth layers, with each layer showing a subset of the segmented regions. The layered representation can be used in a variety of applications. In this paper, the application aims at synthesizing 3-D images for enhanced telepresence allowing the user to look around in natural scenes (intermediate views for interactive displays). In another application, 3-D images showing a natural depth-of-focus are synthesized in order to improve viewing comfort with 3-D displays.

Sprite generation and coding in multiview image sequences

A novel algorithm for the generation of background sprite images from multiview image sequences is presented. A dynamic programming algorithm, first proposed by Grammalidis and Strintzis (see IEEE Trans. Circuits Syst. Video Technol., vol.8, p.328-44, 1998) using a multiview matching cost, as well as pure geometrical constraints, is used to provide an estimate of the disparity field and to identify occluded areas. By combining motion, disparity, and occlusion information, a sprite image corresponding to the first (main) view at the first time instant is generated. Image pixels from other views that are occluded in the main view are also added to the sprite. Finally, the sprite coding method defined by MPEG-4 is extended for multiview image sequences based on the generated sprite. Experimental results are presented, demonstrating the performance of the proposed technique and comparing it with standard MPEG-4 coding methods applied independently to each view.

Flexible 3D motion estimation and tracking for multiview image sequence coding 1 1This work was supported by the EU CEC Project ACTS PANORAMA (Package for New Autostereoscopic Multiview Systems and Applications, ACTS project 092) and the Greek Secretariat for Science and Technology Programme YPER.

This paper describes a procedure for model-based coding of all channels of a multiview image sequence. The 3D model is initialised by accurate adaptation of a 2D wireframe model to the foreground object of one of the views. The rigid 3D motion is estimated for each triangle, and spatial homogeneity neighbourhood constraints are used to improve the reliability of the estimation efficiency and to smooth the motion field produced. A novel technique is used to estimate flexible motion of the nodes of the wireframe from the rigid 3D motion vectors of the wireframe triangles containing each node. Kalman filtering is used to track both rigid 3D motion of each triangle and flexible deformation of each node of the wireframe. The performance of the resulting 3D flexible motion estimation method is evaluated experimentally.