Viewpoint Images Research Articles

Action recognition for depth video using multi-view dynamic images

Dynamic imaging is a recently proposed action description paradigm for simultaneously capturing motion and temporal evolution information, particularly in the context of deep convolutional neural networks (CNNs). Compared with optical flow for motion characterization, dynamic imaging exhibits superior efficiency and compactness. Inspired by the success of dynamic imaging in RGB video, this study extends it to the depth domain. To better exploit three-dimensional (3D) characteristics, multi-view dynamic images are proposed. In particular, the raw depth video is densely projected with respect to different virtual imaging viewpoints by rotating the virtual camera within the 3D space. Subsequently, dynamic images are extracted from the obtained multi-view depth videos and multi-view dynamic images are thus constructed from these images. Accordingly, more view-tolerant visual cues can be involved. A novel CNN model is then proposed to perform feature learning on multi-view dynamic images. Particularly, the dynamic images from different views share the same convolutional layers but correspond to different fully connected layers. This is aimed at enhancing the tuning effectiveness on shallow convolutional layers by alleviating the gradient vanishing problem. Moreover, as the spatial occurrence variation of the actions may impair the CNN, an action proposal approach is also put forth. In experiments, the proposed approach can achieve state-of-the-art performance on three challenging datasets.

Multiview Synthetic Aperture Radar Automatic Target Recognition Optimization: Modeling and Implementation

Multiview synthetic aperture radar (SAR) images could provide much richer information for automatic target recognition (ATR) than from a single-view image. It is desirable to find optimal SAR platform flight paths and acquire a sequence of SAR images from appropriate views, so that multiview SAR ATR can be carried out accurately and efficiently. In this paper, a novel optimization framework for multiview SAR ATR is proposed and implemented. The geometry of the multiview SAR ATR is modeled according to the recognition mission and flight environment. Then, the multiview SAR ATR is abstracted and transformed into a constrained multiobjective optimization problem with objective functions considering the tradeoffs between recognition performance and efficiency and security. A specific approach based on convolutional neural network ensemble and constrained nondominated sorting genetic algorithm II is employed to solve the multiobjective optimization, and optimal flight paths and corresponding imaging viewpoints are obtained. The SAR sensor can thus choose an applicable flight path to acquire the multiview SAR images from different tradeoff solutions according to application requirements. Finally, accurate recognition results can be obtained based on those multiview SAR images. Extensive experiments have shown the validity and superiority of the proposed optimization framework of multiview SAR ATR.

Learning to Predict 3D Surfaces of Sculptures from Single and Multiple Views

The objective of this work is to reconstruct the 3D surfaces of sculptures from one or more images using a view-dependent representation. To this end, we train a network, SiDeNet, to predict the Silhouette and Depth of the surface given a variable number of images; the silhouette is predicted at a different viewpoint from the inputs (e.g. from the side), while the depth is predicted at the viewpoint of the input images. This has three benefits. First, the network learns a representation of shape beyond that of a single viewpoint, as the silhouette forces it to respect the visual hull, and the depth image forces it to predict concavities (which don’t appear on the visual hull). Second, as the network learns about 3D using the proxy tasks of predicting depth and silhouette images, it is not limited by the resolution of the 3D representation. Finally, using a view-dependent representation (e.g. additionally encoding the viewpoint with the input image) improves the network’s generalisability to unseen objects. Additionally, the network is able to handle the input views in a flexible manner. First, it can ingest a different number of views during training and testing, and it is shown that the reconstruction performance improves as additional views are added at test-time. Second, the additional views do not need to be photometrically consistent. The network is trained and evaluated on two synthetic datasets—a realistic sculpture dataset (SketchFab), and ShapeNet. The design of the network is validated by comparing to state of the art methods for a set of tasks. It is shown that (i) passing the input viewpoint (i.e. using a view-dependent representation) improves the network’s generalisability at test time. (ii) Predicting depth/silhouette images allows for higher quality predictions in 2D, as the network is not limited by the chosen latent 3D representation. (iii) On both datasets the method of combining views in a global manner performs better than a local method. Finally, we show that the trained network generalizes to real images, and probe how the network has encoded the latent 3D shape.

Digital blind watermarking based on depth variation prediction map and DWT for DIBR free-viewpoint image

The free-view image could suffer from a viewpoint change attack, although it may be not malicious. This paper focuses not only on this viewpoint change attack, but also on the other traditional and the geometric attacks. We propose a digital watermarking scheme to protect the ownership of a free-view 2D or 3D image, such that the viewer watches the image(s) by rendering an arbitrary viewpoint image (or multiple images) with the received texture image and its depth image. First, the proposed scheme generates a special map, termed the depth variation prediction map (DVPM), to find locations that are safe from viewpoint change. A 3-level Mallat-tree 2-dimensional discrete wavelet transform (2D DWT) is also used, from which the three horizontally low-pass filtered and vertically high-pass filtered subbands are used as the watermark embedding regions, in conjunction with the DVPM. Next, Multiples of the watermark data are embedded into the three subbands, and each watermark bit is embedded into a 2D DWT coefficient by a quantization index modulation (QIM) method, where the quantization step is decided by considering the energy of each subband. In extracting the watermark data, the multiple watermark data are extracted, and the most frequent value in each bit position is taken from the extracted data to form the final data.We experiment the proposed method with various test images for the various attacks, and compare our scheme with the previous methods, to show that the proposed method has excellent performance.

Light-Field Rendering In the View Interpolation Region without Dense Light-Field Reconstruction

Different scenes according to a user’s movement are important to provide a real three-dimensional (3D) experiment. The different scenes, called virtual viewpoint images, can be rendered by 3D structures. However, 3D structure reconstruction from camera images requires high computational complexity and computation time. A light-field rendering algorithm uses the light fields captured by the camera array as the input and selects the light rays for the virtual viewpoint image. Herein, a light-field rendering algorithm in the view interpolation region using line interpolation is introduced. The proposed algorithm functions well to generate planar viewpoint images and zoomed in/out viewpoint images.

A Group of Viewpoint Changes in the Conformal Clifford Algebra

In this paper, we propose a conformal Lie group modeling of the viewpoint changes in the conformal Clifford algebra. Our approach is based on efficient representations of every image viewpoint in the Minkowski space of dimension 4. This enables us to linearize the viewpoint change by a conformal linear transformation of $$\mathbb {R}^{3,1}$$ . We describe the action of our conformal group on the set of images and encode every planar object by an orbit of its images from all viewpoints.

Vision-based Bed Detection for Hospital Patient Monitoring System.

In recent years, as a way to prevent patient fall-down, studies have been conducted using patient room cameras to detect the patient behavior of leaving the bed. It is very important to specify the patient bed location in the process of detecting patient behavior using camera images. In this study, we propose a method to specify the patient bed location using a monocular camera. In this proposal, we convert a camera image viewpoint into a bird's-eye view image as a preprocessing step. By using planer perspective transformation, it is possible to display the bed as a rectangular shape with a fixed ratio, even if the bed location or camera position is changed. Therefore, it is possible to detect the bed location with a high degree of accuracy by means of machine learning. The simulation experiment results confirm that the average error and standard deviation of the bed coordinates are 7.9 and 5.0 pixels, respectively; in the practical scene, we confirm that the average error and standard deviation of the bed coordinates are 12.1 and 8.2 pixels, respectively.

Three-dimensional holoscopic image-coding scheme using a sparse viewpoint image array and disparities

Three-dimensional (3-D) holoscopic imaging is a candidate promising 3-D technology that can overcome some drawbacks of current 3-D technologies. Due to the particular optical structure, a holoscopic image consists of an array of two-dimensional microimages (MIs) that represent different perspectives of the scene. To address the data-intensive characteristics and specific structure of holoscopic images, efficient coding schemes are of utmost importance for efficient storage and transmission. We propose a 3-D holoscopic image-coding scheme using a sparse viewpoint image (VI) array and disparities. In the proposed scheme, a holoscopic image is decomposed into a VI array totally and the VI array is sampled into a sparse VI array. To reconstruct the full holoscopic image, disparities between adjoining MIs are calculated. Based on the remainder set of VIs and disparities, a full holoscopic image is reconstructed and encoded as a reference frame for the coding of the full holoscopic image. As an outcome of the representation, we propose a multiview plus depth compression scheme for 3-D holoscopic images coding. Experimental results show that the proposed coding scheme can achieve an average of 51% bit-rate reduction compared with high efficiency video coding intracoding.

Elemental image array generation method by using optimized depth image‐based rendering algorithm for integral imaging display

AbstractWe propose an elemental image array (EIA) generation method by using an optimized depth image‐based rendering (DIBR) algorithm. In this method, the EIA is synthesized by the reference and virtual viewpoint elemental images, and the virtual viewpoint elemental images at the given locations are generated by DIBR algorithm. We optimize the existing DIBR algorithm by adaptively repairing the warped depth images in the processing part and extend the generation dimension of the virtual viewpoint elemental images from one dimension to two dimensions. The optimized DIBR algorithm can effectively solve the problem: the low quality of virtual viewpoint elemental images caused by discontinuous depth values and disocclusion regions. We also implement the generations of virtual viewpoint elemental images and EIA in graph processing unit to reduce the time cost. Experimental results show that the proposed method can not only improve the quality of the virtual viewpoint images but also accelerate the generations of the virtual viewpoint elemental images and EIA.

21‐1: Reducing Image Quality Variation with Motion Parallax for Glassless 3D Screens using Linear Blending Technology

The linear blending of viewing zones is effective for reducing the number of projectors of a multi‐view 3D screen. However, the image quality of intermediate viewpoint degrades because the adjacent viewpoint's images overlap. We propose a method to generate the viewpoint images for linear blending by adding a phase‐based disparity. By image quality evaluation, we show that our method reduces image quality variation.

Multi-viewpoint Image Array Virtual Viewpoint Rapid Generation Algorithm Based on Image Layering

The use of multi-view image array combined with virtual viewpoint generation technology to record 3D scene information in large scenes has become one of the key technologies for the development of integrated imaging. This paper presents a virtual viewpoint rendering method based on image layering algorithm. Firstly, the depth information of reference viewpoint image is quickly obtained. During this process, SAD is chosen as the similarity measure function. Then layer the reference image and calculate the parallax based on the depth information. Through the relative distance between the virtual viewpoint and the reference viewpoint, the image layers are weighted and panned. Finally the virtual viewpoint image is rendered layer by layer according to the distance between the image layers and the viewer. This method avoids the disadvantages of the algorithm DIBR, such as high-precision requirements of depth map and complex mapping operations. Experiments show that, this algorithm can achieve the synthesis of virtual viewpoints in any position within 2×2 viewpoints range, and the rendering speed is also very impressive. The average result proved that this method can get satisfactory image quality. The average SSIM value of the results relative to real viewpoint images can reaches 0.9525, the PSNR value can reaches 38.353 and the image histogram similarity can reaches 93.77%.

2D–3D registration for cranial radiation therapy using a 3D kV CBCT and a single limited field‐of‐view 2D kV radiograph

We present and evaluate a fully automated 2D-3D intensity-based registration framework using a single limited field-of-view (FOV) 2D kV radiograph and a 3D kV CBCT for 3D estimation of patient setup errors during brain radiotherapy. We evaluated two similarity measures, the Pearson correlation coefficient on image intensity values (ICC) and maximum likelihood measure with Gaussian noise (MLG), derived from the statistics of transmission images. Pose determination experiments were conducted on 2D kV radiographs in the anterior-posterior (AP) and left lateral (LL) views and 3D kV CBCTs of an anthropomorphic head phantom. In order to minimize radiation exposure and exclude nonrigid structures from the registration, limited FOV 2D kV radiographs were employed. A spatial frequency band useful for the 2D-3D registration was identified from the bone-to-no-bone spectral ratio (BNBSR) of digitally reconstructed radiographs (DRRs) computed from the 3D kV planning CT of the phantom. The images being registered were filtered accordingly prior to computation of the similarity measures. We evaluated the registration accuracy achievable with a single 2D kV radiograph and with the registration results from the AP and LL views combined. We also compared the performance of the 2D-3D registration solutions proposed to that of a commercial 3D-3D registration algorithm, which used the entire skull for the registration. The ground truth was determined from markers affixed to the phantom and visible in the CBCT images. The accuracy of the 2D-3D registration solutions, as quantified by the root mean squared value of the target registration error (TRE) calculated over a radius of 3 cm for all poses tested, was ICCAP : 0.56 mm, MLGAP : 0.74 mm, ICCLL : 0.57 mm, MLGLL : 0.54 mm, ICC (AP and LL combined): 0.19 mm, and MLG (AP and LL combined): 0.21 mm. The accuracy of the 3D-3D registration algorithm was 0.27 mm. There was no significant difference in mean TRE for the 2D-3D registration algorithms using a single 2D kV radiograph with similarity measure and image view point. There was no significant difference in mean TRE between ICCLL , MLGLL , ICC (AP and LL combined), MLG (AP and LL combined), and the 3D-3D registration algorithm despite the smaller FOV used for the 2D-3D registration. While submillimeter registration accuracy was obtained with both ICC and MLG using a single 2D kV radiograph, combining the results from the two projection views resulted in a significantly smaller (P≤0.05) mean TRE. Our results indicate that it is possible to achieve submillimeter registration accuracy with both ICC and MLG using either single or dual limited FOV 2D kV radiographs of the head in the AP and LL views. The registration accuracy suggests that the 2D-3D registration solutions presented are suitable for the estimation of patient setup errors not only during conventional brain radiation therapy, but also during stereotactic procedures and proton radiation therapy where tighter setup margins are required.

BoxCars: Improving Fine-Grained Recognition of Vehicles Using 3-D Bounding Boxes in Traffic Surveillance

In this paper, we focus on fine-grained recognition of vehicles mainly in traffic surveillance applications. We propose an approach that is orthogonal to recent advancements in fine-grained recognition (automatic part discovery and bilinear pooling). In addition, in contrast to other methods focused on fine-grained recognition of vehicles, we do not limit ourselves to a frontal/rear viewpoint, but allow the vehicles to be seen from any viewpoint. Our approach is based on 3-D bounding boxes built around the vehicles. The bounding box can be automatically constructed from traffic surveillance data. For scenarios where it is not possible to use precise construction, we propose a method for an estimation of the 3-D bounding box. The 3-D bounding box is used to normalize the image viewpoint by "unpacking" the image into a plane. We also propose to randomly alter the color of the image and add a rectangle with random noise to a random position in the image during the training of convolutional neural networks (CNNs). We have collected a large fine-grained vehicle data set BoxCars116k, with 116k images of vehicles from various viewpoints taken by numerous surveillance cameras. We performed a number of experiments, which show that our proposed method significantly improves CNN classification accuracy (the accuracy is increased by up to 12% points and the error is reduced by up to 50% compared with CNNs without the proposed modifications). We also show that our method outperforms the state-of-the-art methods for fine-grained recognition.

Efficient Light Field Images Compression Method Based on Depth Estimation and Optimization

Massive recent investigations from both industry and academia have been poured into autostereoscopic display. One of the emerging techniques, light field image (LFI), can provide more immersive perception by increasing the number of views and the spatial resolution. However, these advantages restrict the storage and transmission due to such dense-view image simultaneously. To solve this problem, we propose to compress the LFI using multiview video plus depth (MVD) coding architecture. In this paper, we preliminarily estimate the depth based on the concept of epipolar plane image. To achieve a depth value, we design an optimal slope decision algorithm to determine the best slope with the minimal cost. While this rough estimation produces some error points within initial depth map, therefore, we present a depth optimization algorithm using the characteristic of the associated texture image. Ultimately, a small number of selected viewpoint images are encoded with their corresponding depth maps using the MVD framework, and then, the unselected viewpoint images are synthesized by depth image-based rendering technique. To verify the validity of the proposed LFI compression scheme, extensive experiments are conducted. The simulated results demonstrate that the proposed depth map estimation algorithm is superior to other state-of-the-art methods for the LFI. Meanwhile, our LFI compression method outperforms other LFI compression algorithms significantly.

Another View Point Image Generation with Shifted Perspective Using Two In-Vehicle Camera Images for Car Navigation System Based on Vehicular Ad-hoc Networks

It is expected that telepresence techniques are introduced into a car navigation system. For example, the navigation system presents a road scene taken from different points of view to drivers. The realization of the system require cameras on a road demanded by drivers. However, available cameras of roads are generally insufficient. We aim to construct an another view point image generation with shifted perspective using two in-vehicle camera images based on ad-hoc networks. Using the models of a few virtual planes produced from measurement points, we have shown that the transformation is performed in real time. Experimental results demonstrated the effectiveness of the proposed method.

A Fast Orientation Invariant Detector Based on the One-stage Method

We propose an object detection method that predicts the orientation bounding boxes (OBB) to estimate objects locations, scales and orientations based on YOLO (You Only Look Once), which is one of the top detection algorithms performing well both in accuracy and speed. Horizontal bounding boxes(HBB), which are not robust to orientation variances, are used in the existing object detection methods to detect targets. The proposed orientation invariant YOLO (OIYOLO) detector can effectively deal with the bird’s eye viewpoint images where the orientation angles of the objects are arbitrary. In order to estimate the rotated angle of objects, we design a new angle loss function. Therefore, the training of OIYOLO forces the network to learn the annotated orientation angle of objects, making OIYOLO orientation invariances. The proposed approach that predicts OBB can be applied in other detection frameworks. In additional, to evaluate the proposed OIYOLO detector, we create an UAV-DAHUA datasets that annotated with objects locations, scales and orientation angles accurately. Extensive experiments conducted on UAV-DAHUA and DOTA datasets demonstrate that OIYOLO achieves state-of-the-art detection performance with high efficiency comparing with the baseline YOLO algorithms.

A Work Area Visualization by Multi-View Camera-Based Diminished Reality

Hand-held tools are indispensable for efficient manual working in fields ranging from woodworking to surgery. In this paper, we present a diminished reality (DR) method to visualize areas occluded by hands and tools in various hand-working scenarios. We propose a redesigned existing arbitrary viewpoint image generation method for DR applications as a core DR background rendering to recover views without undesirable objects. We conducted quantitative and qualitative experiments using real data to validate the performance of the proposed method. The experimental results showed that our method runs in real-time (40.1 fps) and surpasses conventional methods (including image inpainting-based and geometry-based approaches) in terms of image similarity measures.

Remote Sensing Image Registration Using Multiple Image Features

Remote sensing image registration plays an important role in military and civilian fields, such as natural disaster damage assessment, military damage assessment and ground targets identification, etc. However, due to the ground relief variations and imaging viewpoint changes, non-rigid geometric distortion occurs between remote sensing images with different viewpoint, which further increases the difficulty of remote sensing image registration. To address the problem, we propose a multi-viewpoint remote sensing image registration method which contains the following contributions. (i) A multiple features based finite mixture model is constructed for dealing with different types of image features. (ii) Three features are combined and substituted into the mixture model to form a feature complementation, i.e., the Euclidean distance and shape context are used to measure the similarity of geometric structure, and the SIFT (scale-invariant feature transform) distance which is endowed with the intensity information is used to measure the scale space extrema. (iii) To prevent the ill-posed problem, a geometric constraint term is introduced into the L2E-based energy function for better behaving the non-rigid transformation. We evaluated the performances of the proposed method by three series of remote sensing images obtained from the unmanned aerial vehicle (UAV) and Google Earth, and compared with five state-of-the-art methods where our method shows the best alignments in most cases.

CrossbowCam: a handheld adjustable multi-camera system

This paper presents a novel multi-functional, low-cost handheld multi-camera system (one dimensional camera array) - “CrossbowCam”. The CrossbowCam is suitable for multi-viewpoint image acquisition, smooth switching, alignment and seamless stitching applications. The proposed system differs from the traditional fixed image acquisition systems which are large-sized, high-priced, single functional, and can only captured images at specific locations. With the proposed system, the users can push one single button to change the configuration of the camera array rapidly to divergence (convex arc), parallel (linear), or convergence (concave arc). The three camera configurations can each be suitable for applications such as panorama image stitching, autostereoscopic 3D display, bullet-time (time-freeze) visual effect, 3D scene reconstruction, etc. To rapidly acquire the relationship among cameras after configuration change, we propose a two-stage calibration method to compensate the mechanical misalignment. The first stage adopts the traditional checkerboard calibration method to get the intrinsic parameters (focal length, principal point) and the lens distortion for each camera. The second stage requires no auxiliary tool but utilizes a large number of common feature points from multiple viewpoint images to acquire the extrinsic parameters (translation and rotation matrix) and to compensate the vertical misalignment and the horizontal uneven angle distribution due to the mechanical structure. The proposed system can then insert virtual viewpoint images between actual viewpoint images to allow the viewpoint switching more smoothly. The proposed system has eight cameras with maximum viewing angle of 90° in divergence mode, 38 mm spacing in parallel mode, and imaging radius of 10 m ∼ 0.5 m in convergence mode. We believe that the proposed system can potentially change the consumer habits and becomes the new type of home-use handheld camcorder system in the future.

Human detection in occluded scenes through optically inspired multi-camera image fusion.

In this paper, a novel approach for foreground extraction has been proposed based on a popular three-dimensional imaging technique in optics, called integral imaging. In this approach, multiple viewpoint images captured from a three-dimensional scene are used to extract range information of the scene and effectively extract an object or a person, even in the presence of heavy occlusion. The algorithm consists of two parts: depth estimation and reconstruction of the targeted object at the estimated depth distance. Further processing of the resulting reconstructed image can lead to the detection of a face or a pedestrian in the scene, which may not otherwise be detectable due to partial occlusion in each of the views. The validity of our approach has been demonstrated by experimental results in different scenarios.