Arbitrary Viewpoints Research Articles

Development and testing of an endoscopic pseudo-viewpoint alternating system.

Purpose An endoscopic system is needed that presents informative images irrespective of the surgical situation and the number of degrees of freedom in endoscopic manipulation. This goal may be achieved with a virtual reality view for a region of interest from an arbitrary viewpoint. An endoscopic pseudo-viewpoint alternation system for this purpose was developed and tested.Method Surgical experts and trainees from an endoscopic surgery training course at the minimally invasive surgery training center of Kyushu University were enrolled in a trial of a virtual reality system. The initial viewpoint was positioned to approximate the horizontal view often seen in laparoscopic surgery, with 20^{circ } between the optical axis of the endoscope and the task surface. A right-to-left suturing task with right hand, based on a task from the endoscopic surgery training course, was selected for testing. We compared task outcomes with and without use of a new virtual reality-viewing system.Result There was a 0.37 mm reduction in total error (p = 0.02) with use of the proposed system. Error reduction was composed of 0.1 mm reduction on the y-axis and 0.27 mm reduction on the x-axis. Experts benefited more than novices from use of the proposed system. Most subjects worked at a pseudo-viewpoint of around 34^circ .Discussion Suturing performance improved with the new virtual reality endoscopic display system. Viewpoint alternation resulted in an overview that improved depth perception and allowed subjects to better aim the marker. This suggests the proposed method offers users better visualization and control in endoscopic surgery.

View invariant action recognition using generalized 4D features

In this paper, we propose a method to recognize human actions independently of viewpoints by developing 4D space–time features which can generalize the information from a finite number of views in training phase so as to show a satisfactory performance in arbitrary testing views. This 4D space–time interest points (4D-STIPs, [x,y,z,t]) are extracted using 3D space volumes reconstructed from images of a finite number of different views. Since the proposed features are constructed using volumetric information, the features for arbitrary 2D space viewpoint in testing can be generated by projecting the 3D space volumes and 4D-STIPs on corresponding test image planes. This enables action recognition in any camera viewpoint even after training with images from only a finite number of views. We also propose the variant of 3D space–time interest points, which take into account the simultaneous gradient variation in all 3 dimensions to focus on the motion of important spatial corner points. 3D space volumes and 4D-STIPs can be projected to arbitrary viewpoints for training each action to get generalization capability of the classifier. With these projected features, we construct motion history images and non-motion history images which encode the moving and non-moving parts of an action respectively. After reducing the feature dimension, the final features are learned by support vector data description method. In experiments, we train the models using IXMAS dataset constructed from five views and test them with a new SNU dataset made for evaluating the generalization performance for arbitrary view videos.

55.4: Optimized Parallax Control of Arbitrary Viewpoint Images with Motion Parallax on Autostereoscopic Displays

AbstractWe propose an autostereoscopic display system which optimizes a parallax value of arbitrary viewpoint images to reduce the influence of 3D‐crosstalk space and to display motion parallax. The system monitors the observer's eye positions, and varies binocular parallax based on the viewing angle.

영상의 시점변화와 악의적 공격에 대한 디지털 워터마킹

This paper deals with digital watermarking methods to protect ownership of image with targeting the ultra-high multi-view or free-view image service in which an arbitrary viewpoint image should be rendered at the user side. The main purpose of it is not to propose a superior method to the previous methods but to show how difficult to construct a watermarking scheme to overcome the viewpoint translation attack. Therefore we target the images with various attacks including viewpoint translation. This paper first shows how high the error rate of the extracted watermark data from viewpoint-translated image by basic schemes of the method using 2DDCT(2D discrete cosine transform) and the one using 2DDWT(2D discrete wavelet transform), which are for 2D image. Because the difficulty in watermarking for the viewpoint-translated image comes from the fact that we don`t know the translated viewpoint, we propose a scheme to find the translated viewpoint, which uses the image and the corresponding depth information at the original viewpoint. This method is used to construct the two non-blind watermarking methods to be proposed. They are used to show that recovery of the viewpoint affect a great deal of the error rate of the extracted watermark. Also by comparing the performances of the proposed methods and the previous ones, we show that the proposed ones are better in invisibility and robustness, even if they are non-blind.

깊이 및 텍스쳐 영상 기반의 3D 입체 영상을 위한 워터마킹 알고리즘

3차원 영상을 생성하기 위해 깊이영상과 텍스쳐 영상의 활용이 보편화되면서 이들 영상에 대한 보호도 많은 관심을 받고 있다. 본 논문에서는 깊이와 텍스쳐 영상으로부터 임의의 시점에 생성되는 스테레오 및 다시점 영상의 소유권을 보호하기 위한 워터마킹 기법을 제안한다. 3D 워핑을 이용한 DIBR을 통해서 워터마크가 보존될 수 있도록 대상 영역을 설정한 이후에 2차원 DCT를 통해서 텍스쳐 영상을 주파수 계수로 변환한다. 이 계수의 일부를 양자화하여 워터마크를 삽입한다. 삽입된 워터마크는 확인이 되지 않기 때문에 비가시성을 확인할 수 있었고, 일반적인 영상처리 공격에 의해서 강인함을 확인하였다. Since the depth and texture images have been widely used for generating 3-dimensional stereoscopic image, the security of them have been focused. In this paper, we propose a new watermarking technique for copyright of stereo and multiview images which is generated in an arbitrary viewpoint by depth and texture image. After the mark space is selected for preserving watermark through DIBR (depth-image-based rendering) process which uses 3D warping, the texture image is transformed to the frequency coefficient using 2D DCT (discrete cosine transform). Some parts of them are quantized, which is the corresponding process to watermarking. The embedded watermark is not conformed by eyes, so we identified the invisibility of the proposed method. In case of appling attacks of general image process, we also identified the robustness of it.

Projection Mapping on Arbitrary Cubic Cell Complexes

AbstractThis work presents a new representation used as a rendering primitive of surfaces. Our representation is defined by an arbitrary cubic cell complex: a projection‐based parameterization domain for surfaces where geometry and appearance information are stored as tile textures. This representation is used by our ray casting rendering algorithm called projection mapping, which can be used for rendering geometry and appearance details of surfaces from arbitrary viewpoints. The projection mapping algorithm uses a fragment shader based on linear and binary searches of the relief mapping algorithm. Instead of traditionally rendering the surface, only front faces of our rendering primitive (our arbitrary cubic cell complex) are drawn, and geometry and appearance details of the surface are rendered back by using projection mapping. Alternatively, another method is proposed for mapping appearance information on complex surfaces using our arbitrary cubic cell complexes. In this case, instead of reconstructing the geometry as in projection mapping, the original mesh of a surface is directly passed to the rendering algorithm. This algorithm is applied in the texture mapping of cultural heritage sculptures.

View Management of Projected Labels on Nonplanar and Textured Surfaces

This paper presents a new label layout technique for projection-based augmented reality (AR) that determines the placement of each label directly projected onto an associated physical object with a surface that is normally inappropriate for projection (i.e., nonplanar and textured). Central to our technique is a new legibility estimation method that evaluates how easily people can read projected characters from arbitrary viewpoints. The estimation method relies on the results of a psychophysical study that we conducted to investigate the legibility of projected characters on various types of surfaces that deform their shapes, decrease their contrasts, or cast shadows on them. Our technique computes a label layout by minimizing the energy function using a genetic algorithm (GA). The terms in the function quantitatively evaluate different aspects of the layout quality. Conventional label layout solvers evaluate anchor regions and leader lines. In addition to these evaluations, we design our energy function to deal with the following unique factors, which are inherent in projection-based AR applications: the estimated legibility value and the disconnection of the projected leader line. The results of our subjective experiment showed that the proposed technique could significantly improve the projected label layout.

Real-time estimation of hand gestures based on manifold learning from monocular videos

Object pose estimation by manifold learning has become a hot research area recently. In this paper, we propose an efficient method that can recover pose and viewpoints for numerous hand gestures from monocular videos based on Locality Preserving Projections. We first select some hand dynamic gestures as primitive hand motions and set a 3D-2D mapping table to relate 3D joint angles of sampling static pose with their projective silhouettes from arbitrary viewpoints. Then the embedding space and explicit mapping function are learnt for every primitive motion. In order to make classification and prediction among those embedding spaces, a Subspace Filtering Algorithm is also proposed which can recognize and recover numerous hand dynamic gestures by the combination of primitive gestures. At last, by using skin color cues and oriented k-Dops, multi-hands can be labeled and tracked separately and accurately. Extensive experimental results demonstrate qualitatively and quantitatively that 3D pose recovery of hands can be achieved by our method robustly and efficiently.

Towards precise real-time 3D difference detection for industrial applications

3D difference detection is the task to verify whether the 3D geometry of a real object exactly corresponds to a 3D model of this object. We present an approach for 3D difference detection with a hand-held depth camera. In contrast to previous approaches, with the presented approach geometric differences can be detected in real-time and from arbitrary viewpoints. The 3D difference detection accuracy is improved by two approaches: first, the precision of the depth camera's pose estimation is improved by coupling the depth camera with a high precision industrial measurement arm. Second, the influence of the depth measurement noise is reduced by integrating a 3D surface reconstruction algorithm. The effects of both enhancements are quantified by a ground-truth based quantitative evaluation, both for a time-of-flight (SwissRanger 4000) and a structured light depth camera (Kinect). With the proposed enhancements, differences of few millimeters can be detected from 1m measurement distance.

Visualization of the Workpieces on the CNC Machines Using the Virtual Camera Based on the Sub-Pixel IBR Method

One of the results of the dynamic progress in the field of industrial and applied computer science and mechatronics is the growing interest in usage of the machine vision algorithms and systems. One of such areas of applications is the positioning of the workpieces on the CNC machines based on the image analysis, allowing also the visualization of the workpiece observed from an arbitrary viewpoint located between two or more cameras. Such virtual camera can be effectively implemented using Image Based Rendering technology [ but one of its limitations is the necessity of using the depth maps, which have to be acquired for each reference image. It can be done using the additional fringe patterns projector, similarly as in typical 3D scanning systems or using some other approaches e.g. stereovision. Nevertheless, the quality of images obtained using the typical IBR method is limited by the missing pixels (holes), which should be filled using splatting algorithm, which typically cause the noticeable blurring of obtained image. The solution proposed and discussed in this paper is based on the sub-pixel IBR method with additional filtering and interpolation of the points acquired from the reference cameras allowing increasing the quality of the final images obtained as the visualization of the positioned workpieces.

Viewpoint invariant gender recognition

In this paper, we address a problem of gender classification of faces taken from arbitrary viewpoints. We use a face model for accurate face localisation based on a combination of appearance and geometry. A probabilistic matching of particular traits on face allows to classify the gender of face even when pose changes. We deal with the local invariant features whose performances have already been proved. Each facial feature retained in the detection step will be weighted by a probability to be male or female. Such feature contributes to determine the gender associated to a given face. We evaluate the model by testing it simultaneously in face localisation and gender classification experiments on PIE, FERET and CMU-profiles databases. The experimental results show that the probabilistic invariant model performs well to detect face and gives a rate of 92.1% of accurate gender classification in the presence of viewpoint changes and large appearance variability of faces.

Benchmark 3D Models of Natural Navigation Environments @ www.InsectVision.org

A number of models for insect navigation have been developed to explain the results of behavioural experiments and to be tested in simulations or on robotic platforms. However, it remains difficult to evaluate these models in a generic and objective way, beyond the specific results they have been developed to explain and beyond the specific environments and applications for which they have been developed. Here we introduce a realistic and controllable environment, that allows one to test visual navigation algorithms in natural and complex conditions and that can be shared with the research community. We have launched a publicly accessible service at www.insectvision.org that allows users to render panoramic views at any position within 3D models of natural habitats and to filter them with the sampling array of honeybee compound eyes. Using 3D laser scans and multi-camera images, we generate 3D models of environments (see accompanying abstract “Mapping the Navigational Information Content of Insect Habitats” by Sturzl et al.). These models allow us to render views of the scene from arbitrary viewpoints within the acquired workspace. We can then move a virtual insect, in our case a bee, through this environment along any path and reconstruct a view from the cockpit of the insect, including a model of its compound eye sampling array, if available (see Fig. 1). This not only allows us to investigate visual navigation mechanisms in insects, but also the performance of specific models of navigation under real-life conditions. The currently available 3D model is based on data acquired in the habitat of ground nesting wasps and bees. We recorded learning and homing flights, which are also available online. We see this work as the first step towards a growing database that can be used for comparative analysis and for rigorous tests of insect navigation models. The platform can also be used for benchmarking navigation and flight-control models in biology and robotics.

Novel‐View Synthesis of Outdoor Sport Events Using an Adaptive View‐Dependent Geometry

AbstractWe propose a novel fully automatic method for novel‐viewpoint synthesis. Our method robustly handles multi‐camera setups featuring wide‐baselines in an uncontrolled environment. In a first step, robust and sparse point correspondences are found based on an extension of the Daisy features [TLF10]. These correspondences together with back‐projection errors are used to drive a novel adaptive coarse to fine reconstruction method, allowing to approximate detailed geometry while avoiding an extreme triangle count.To render the scene from arbitrary viewpoints we use a view‐dependent blending of color information in combination with a view‐dependent geometry morph. The view‐dependent geometry compensates for misalignments caused by calibration errors. We demonstrate that our method works well under arbitrary lighting conditions with as little as two cameras featuring wide‐baselines. The footage taken from real sports broadcast events contains fine geometric structures, which result in nice novel‐viewpoint renderings despite of the low resolution in the images.

MoPaD--Moment Tensor Plotting and Decomposition: A Tool for Graphical and Numerical Analysis of Seismic Moment Tensors

Online material: Software source code; examples; sample plots. The graphical representation of a focal mechanism as a plot of the radiation pattern on the focal sphere (sometimes referred to as “beachball diagram”) is well established. It consists of a spherical projection on a unit sphere centered at the source point of a seismic event. The sign of the P -wave radiation pattern is shown as colored areas on the sphere (as described, e.g. , by Udias 1999; Stein and Wysession 2003; and Crotin 2004). For pure shear motion on a plane, the focal sphere may be fully described by the angles of strike, dip, and slip-rake of this fault plane ( cf. , Shearer 1999; Aki and Richards 2002). In this case the construction of the spherical projection into the R 2 is straightforward ( e.g. , Stein and Wysession 2003, chapter 5). Several tools are available to do this work. They exist either as a more or less configurable, independent software package (Snoke 2003; Srivastava et al. 2006; Luis 2007; Haeussler and Labay 2007), as supplements for GMT, MATLAB (Creager 1997; Center for Earthquake Research and Information 2008) or Mathematica (Scherbaum et al. 2010), or as an online applet (Helffrich 2007). Almost all of the existing software consider only these pure shear cracks. The visualisation of a more complex source mechanism, e.g. , an arbitrary seismic moment tensor with six independent entries, is often not possible. Only a few tools exist at all that can handle this general case. The most commonly used one is the psmeca -tool in GMT (Wessel and Smith 1999). However, projections for arbitrary viewpoints are not possible with psmeca . We present the free platform-independent command line tool MoPaD , written in the Python programming language that allows one …

스테레오 깊이영상의 변위증분을 이용한 중간시점 깊이영상 생성

This paper proposes a method to generate a depth image at an arbitrary intermediate view-point, which is targeting a video service for free-view, auto-stereoscopy, etc. It assumes that the leftmost and the rightmost depth images are given and they both have been camera-calibrated and image-rectified. This method calculates and uses a disparity increment per depth value. In this paper, it is obtained by stereo matching for the given two depth image by considering more general cases. The disparity increment is used to find the location in the intermediate view-point depth image (IVPD) for each depth in the given images. Thus, this paper finds two IVPDs, from left image and from right image. Noises are removed and holes are filled in each IVPDs and the two results are combined to get the final IVPD. The proposed method was implemented and applied to several test sequences. The results revealed that the quality of the generated IVPD corresponds to 33.84dB of PSNR in average and it takes about 1 second to generate a HD IVPD. We evaluate that this image quality is quite good by considering the low correspondency among the left images, intermediate images, and the right images in the test sequences. If the execution speed is improved, the proposed method can be a very useful method to generate an IVPD at an arbitrary view-point, we believe.

Autocalibration of Multiprojector CAVE-Like Immersive Environments

In this paper, we present the first method for the geometric autocalibration of multiple projectors on a set of CAVE-like immersive display surfaces including truncated domes and 4 or 5-wall CAVEs (three side walls, floor, and/or ceiling). All such surfaces can be categorized as swept surfaces and multiple projectors can be registered on them using a single uncalibrated camera without using any physical markers on the surface. Our method can also handle nonlinear distortion in the projectors, common in compact setups where a short throw lens is mounted on each projector. Further, when the whole swept surface is not visible from a single camera view, we can register the projectors using multiple pan and tilted views of the same camera. Thus, our method scales well with different size and resolution of the display. Since we recover the 3D shape of the display, we can achieve registration that is correct from any arbitrary viewpoint appropriate for head-tracked single-user virtual reality systems. We can also achieve wallpapered registration, more appropriate for multiuser collaborative explorations. Though much more immersive than common surfaces like planes and cylinders, general swept surfaces are used today only for niche display environments. Even the more popular 4 or 5-wall CAVE is treated as a piecewise planar surface for calibration purposes and hence projectors are not allowed to be overlapped across the corners. Our method opens up the possibility of using such swept surfaces to create more immersive VR systems without compromising the simplicity of having a completely automatic calibration technique. Such calibration allows completely arbitrary positioning of the projectors in a 5-wall CAVE, without respecting the corners.

Human tooth pulp anatomy visualization by 3D magnetic resonance microscopy

BackgroundPrecise assessment of dental pulp anatomy is of an extreme importance for a successful endodontic treatment. As standard radiographs of teeth provide very limited information on dental pulp anatomy, more capable methods are highly appreciated. One of these is 3D magnetic resonance (MR) microscopy of which diagnostic capabilities in terms of a better dental pulp anatomy assessment were evaluated in the study.Materials and methodsTwenty extracted human teeth were scanned on a 2.35 T MRI system for MR microscopy using the 3D spin-echo method that enabled image acquisition with isotropic resolution of 100 μm. The 3D images were then post processed by ImageJ program (NIH) to obtain advanced volume rendered views of dental pulps.ResultsMR microscopy at 2.35 T provided accurate data on dental pulp anatomy in vitro. The data were presented as a sequence of thin 2D slices through the pulp in various orientations or as volume rendered 3D images reconstructed form arbitrary view-points. Sequential 2D images enabled only an approximate assessment of the pulp, while volume rendered 3D images were more precise in visualization of pulp anatomy and clearly showed pulp diverticles, number of pulp canals and root canal anastomosis.ConclusionsThis in vitro study demonstrated that MR microscopy could provide very accurate 3D visualization of dental pulp anatomy. A possible future application of the method in vivo may be of a great importance for the endodontic treatment.

Video-Based In Situ Tagging on Mobile Phones

We propose a novel way to augment a real-world scene with minimal user intervention on a mobile phone; the user only has to point the phone camera to the desired location of the augmentation. Our method is valid for horizontal or vertical surfaces only, but this is not a restriction in practice in manmade environments, and it avoids going through any reconstruction of the 3-D scene, which is still a delicate process on a resource-limited system like a mobile phone. Our approach is inspired by recent work on perspective patch recognition, but we adapt it for better performances on mobile phones. We reduce user interaction with real scenes by exploiting the phone accelerometers to relax the need for fronto-parallel views. As a result, we can learn a planar target in situ from arbitrary viewpoints and augment it with virtual objects in real-time on a mobile phone.

Triangulation of cubic panorama for view synthesis

An unstructured triangulation approach, new to our knowledge, is proposed to apply triangular meshes for representing and rendering a scene on a cubic panorama (CP). It sophisticatedly converts a complicated three-dimensional triangulation into a simple three-step triangulation. First, a two-dimensional Delaunay triangulation is individually carried out on each face. Second, an improved polygonal triangulation is implemented in the intermediate regions of each of two faces. Third, a cobweblike triangulation is designed for the remaining intermediate regions after unfolding four faces to the top/bottom face. Since the last two steps well solve the boundary problem arising from cube edges, the triangulation with irregular-distribution feature points is implemented in a CP as a whole. The triangular meshes can be warped from multiple reference CPs onto an arbitrary viewpoint by face-to-face homography transformations. The experiments indicate that the proposed triangulation approach provides a good modeling for the scene with photorealistic rendered CPs.

Automatic Registration of Multi‐Projector Domes Using a Single Uncalibrated Camera

AbstractIn this paper we present a novel technique for easily calibrating multiple casually aligned projectors on spherical domes using a single uncalibrated camera. Using the prior knowledge of the display surface being a dome, we can estimate the camera intrinsic and extrinsic parameters and the projector to display surface correspondences automatically using a set of images. These images include the image of the dome itself and a projected pattern from each projector. Using these correspondences we can register images from the multiple projectors on the dome. Further, we can register displays which are not entirely visible in a single camera view using multiple pan and tilted views of an uncalibrated camera making our method suitable for displays of different size and resolution. We can register images from any arbitrary viewpoint making it appropriate for a single head‐tracked user in a 3D visualization system. Also, we can use several cartographic mapping techniques to register images in a manner that is appropriate for multi‐user visualization.Domes are known to produce a tremendous sense of immersion and presence in visualization systems. Yet, till date, there exists no easy way to register multiple projectors on a dome to create a high‐resolution realistic visualizations. To the best of our knowledge, this is the first method that can achieve accurate geometric registration of multiple projectors on a dome simply and automatically using a single uncalibrated camera.