Viewpoint Images Research Articles

Free viewpoint image generation system using fisheye cameras and a laser rangefinder for indoor robot teleoperation

In robot teleoperation, a lack of depth information often results in collisions between the robots and obstacles in its path or surroundings. To address this issue, free viewpoint images can greatly benefit the operators in terms of collision avoidance as the operators are able to view the robot’s surrounding from the images at arbitrary points, giving them a better depth information. In this paper, a novel free viewpoint image generation system is proposed. One approach to generate free viewpoint images is to use multiple cameras and Light Detection and Ranging (LiDAR). Instead of using the expensive LiDAR, this study utilizes a cost-effective laser rangefinder (LRF) and a characteristic of man-made environments. In other words, we install multiple fisheye cameras and an LRF on a robot. Free viewpoint images are generated under the assumption that walls are perpendicular to the floor. Furthermore, an easy calibration for estimating the poses of the multiple fisheye cameras, the LRF, and the robot model is proposed. Experimental results show that the proposed method can generate free viewpoint images using cameras and an LRF. Finally, the proposed method is primarily implemented using OpenGL Shading Language to utilize a graphics processing unit computation to achieve a real-time processing of the multiple high-resolution images. Supplementary videos and our source code are available at our project page (https://matsuren.github.io/fvp).

Robust node detection and tracking in fruit-vegetable crops using deep learning and multi-view imaging

Obtaining high-quality phenotypic data that can be used to study the relationship between genotype, phenotype and environment is still labour-intensive. Digital plant phenotyping can assist in collecting these data by replacing human vision by computer vision. However, for complex traits, such as plant architecture, robust and generic digital phenotyping methods have not yet been developed. This study focusses on internode length in cucumber plants. A method for estimating internode length and internode development over time is proposed. The proposed method firstly applies a robust node-detection algorithm based on a deep convolutional neural network. In tests, the algorithm had a precision of 0.95 and a recall of 0.92. The nodes are detected in images from multiple viewpoints around the plant in order to deal with the complex and cluttered plant environment and to solve the occlusion of nodes by other plant parts. The nodes detected in the multiple viewpoint images are then clustered using affinity propagation. The predicted clusters had a homogeneity of 0.98 and a completeness of 0.99. Finally, a linear function is fitted, which allows to study internode development over time. The presented method was able to measure internode length in cucumber plants with a higher accuracy and a larger temporal resolution than other methods proposed in literature and without the time investment needed to obtain the measurements manually. The relative error of our complete method was 5.8%. The proposed method provides many opportunities for robust phenotyping of fruit-vegetable crops grown under greenhouse conditions.

Full-parallax 3D display using time-multiplexing projection technology

We enhanced the resolution characteristics of a threedimensional (3D) image using time-division multiplexing methods in a full-parallax multi-view 3D display. A time-division light-ray shifting (TDLS) method is proposed that uses two polarization gratings (PGs). As PG changes the diffraction direction of light rays according to the polarization state of the incident light, this method can shift light rays approximately 7 mm in a diagonal direction by switching the polarization state of incident light and adjusting the distance between the PGs. We verified the effect on the characteristics of 3D images based on the extent of the shift. As a result, the resolution of a 3D image with depth is improved by shifting half a pitch of a multi-view image using the TDLS method, and the resolution of the image displayed near the screen is improved by shifting half a pixel of each viewpoint image with a wobbling method. These methods can easily enhance 3D characteristics with a small number of projectors.

Optical design for transfer of camera viewpoint using retrotransmissive optical system

We propose a virtual camera that can pass through a small hole in an obstruction to capture an image on the other side. Recently, free-viewpoint television technology has enabled the generation of video in which viewpoint images are captured from locations where cameras are not actually placed. However, capturing the images of objects concealed behind obstructions or beyond a camera’s field of view is a challenge. We designed an optical transformation system that utilizes a conventional camera, concave lens, and transmissive mirror device (TMD); this system enables the capture of images through small holes in walls or other obstructions. Our experimental prototype demonstrated that it is possible to capture images of the area on the other side of a wall through a 5 mm hole. In this paper, modulation transfer function (MTF) comparison is used to show that a combination of a concave lens and TMD is an effective optical design for a midair camera.

Artificial Intelligence Recognition Simulation of 3D Multimedia Visual Image Based on Sparse Representation Algorithm

With the rapid development of computer networks and multimedia technologies, images, which are important carriers of information dissemination, have made human cognition of things easier. Image recognition is a basic research task in computer vision, multimedia search, image understanding and other fields. This paper proposes a hierarchical feature learning structure that is completely automatically based on the original pixels of the image, and uses the K-SVD (K-Singular Value Decomposition) algorithm with label consistency constraints to train the discriminant dictionary. For different types of image data sets, the algorithm only extracts image blocks. After dense sampling, an efficient OMP (Orthogonal Matching Pursuit) encoder is used to obtain a layered sparse representation. The improved SIFT (Scale Invariant Feature Transform) algorithm is used to solve the difficult problem of multimedia visual image stereo matching. The feature point extraction and stereo matching of multimedia visual images, different scales and different viewpoint images are analyzed separately. Aiming at a large number of low-dimensional geometric features of 3D images, this paper studies the extraction and sorting strategies of low-dimensional geometric features of 3D images. A sparse representation method for 3D images is proposed, and the sparseness of image features is evaluated. This further improves the accuracy of 3D image representation and the robustness of 3D image recognition algorithms.

Nighttime Data Augmentation Using GAN for Improving Blind-Spot Detection

Camera-based blind-spot detection systems improve the shortcomings of radar-based systems for accurately detecting the position of a vehicle. However, as with many camera-based applications, the detection performance is insufficient in a low-illumination environment such as at night. This problem can be solved with augmented nighttime images in the training data but acquiring them and annotating the additional images are cumbersome tasks. Therefore, we propose a framework that converts daytime images into synthetic nighttime images using a generative adversarial network and that augments the synthetic images for the training process of the vehicle detector. A public dataset comprising different viewpoints of target images was used to easily obtain the images required for training the generative adversarial network. Experiments on a real nighttime dataset demonstrate that the proposed framework improved the detection performance considerably in comparison with using daytime images only.

Tensor-Based Light Field Compressed Sensing and Epipolar Plane Images Reconstruction via Deep Learning

Light field (LF) can capture the spatial and angular information of the light in one single exposure. And the LF images are widely used in various fields, especially in immersive media. The rich imaging information in the LF poses great challenges for transmission. However, LF images are sparse and redundant to some extent, which makes LF compression possible. Besides, the compressed sensing (CS) theory shows that images can be recovered from a small number of measurements when they are sparse. In this paper, we propose a Tensor-based Compressed Sensing method to compress images and Epipolar Plane Images for reconstruction (TCSEPI). This method divides the viewpoints of LF images into several regions and stacks the images in each region into a 4D tensor, which conduct CS together and yields measurements with common characteristics. Subsequently, the epipolar plane images are used to reconstruct the LF images and restore the geometric consistency information. To achieve better reconstruction results, we design two cascaded convolutional neural networks to implement the measurement matrix optimization and LF images reconstruction sequentially. Experimental results show the superior performance of TCSEPI, which achieves at least 3dB gain in PSNR and outperforms state-of-the-art in the reconstruction quality.

A Real-Time Super Multiview Rendering Pipeline for Wide Viewing-Angle and High-Resolution 3D Displays Based on a Hybrid Rendering Technique

Super multiview (SMV) 3D displays with a directional diffuser can provide dense viewpoints, wide viewing angles and high image quality. However, traditional SMV 3D rendering pipelines have many disadvantages, such as high time costs, low image quality, incorrect lighting effects and limited viewing angles. To overcome the abovementioned drawbacks, we propose a new SMV 3D rendering pipeline based on a hybrid rendering technique (HRT) that effectively combines and improves the Turing multiview rendering technique, the multiple view plus depth (MVD) technique and the deferred shading technique, introducing additional normal information, diffuseness information and shininess information into the SMV 3D rendering pipeline. The SMV HRT pipeline can generate SMV 3D images with accurate lighting in real time when the viewpoint number of an SMV 3D display is greater than 50, the viewing angle is greater than 100 degrees, the resolution of a single viewpoint image is more than 512*512 and the resolution of the LCD panel is 7680*4320; in particular, complex scenes can be generated in real time. The proposed HRT pipeline is the only viable real-time rendering pipeline with accurate lighting effects for super multiview 3D displays with wide viewing angle and high resolution that runs on a common PC with standard graphics hardware. An 8k SMV 3D display with an 80° viewing angle and 100 viewpoints is applied to demonstrate the validity of the HRT method in experiments. When the number of reference views is four, the experimental results show that the rendering frame rate is more than 35 fps on average for complex virtual scenes, and the peak signal-to-noise ratio (PSNR) value of the HRT is greater than 36 dB, illustrating its high image quality and accurate lighting. Compared with state-of-the-art MVD methods, HRT processes more lighting information and is more suitable for virtual scene rendering. Therefore, the HRT method has advantages regarding the image quality of the reconstructed 3D image and can easily achieve real-time rendering for SMV 3D displays with wide viewing angles.

3D Point Cloud Denoising Using Graph Laplacian Regularization of a Low Dimensional Manifold Model.

3D point cloud-a new signal representation of volumetric objects-is a discrete collection of triples marking exterior object surface locations in 3D space. Conventional imperfect acquisition processes of 3D point cloud-e.g., stereo-matching from multiple viewpoint images or depth data acquired directly from active light sensors-imply non-negligible noise in the data. In this paper, we extend a previously proposed low-dimensional manifold model for the image patches to surface patches in the point cloud, and seek self-similar patches to denoise them simultaneously using the patch manifold prior. Due to discrete observations of the patches on the manifold, we approximate the manifold dimension computation defined in the continuous domain with a patch-based graph Laplacian regularizer, and propose a new discrete patch distance measure to quantify the similarity between two same-sized surface patches for graph construction that is robust to noise. We show that our graph Laplacian regularizer leads to speedy implementation and has desirable numerical stability properties given its natural graph spectral interpretation. Extensive simulation results show that our proposed denoising scheme outperforms state-of-the-art methods in objective metrics and better preserves visually salient structural features like edges.

A Bernoulli–Gaussian Binary Inversion Method for High-Frequency Electromagnetic Imaging of Metallic Reflectors

High-frequency electromagnetic inverse scattering of conductors has many practical applications. Its goal is to reconstruct the shape of conductors from their scattered fields at some distances. When viewed from the pixelated imaging viewpoint, any pixel in the imaging domain is either a conductor or background (air); thus such an inverse scattering problem has binary solutions for all pixels. Under the high-frequency physical optics (PO) approximation, the scattered field is formulated as a linear problem in terms of a local binary shape function. A binary inversion method is proposed for the detection and shape reconstruction of a flat PEC reflector from high-frequency scattered field measurements. To exploit the underlying binary structure, a Bernoulli–Gaussian (BG) prior model is employed to perform a binary enforcement mechanism. This binary enforcement mechanism can push each pixel of the inversion domain to have a value equal to either zero or one. Then, the damped generalized approximate message passing (GAMP) is integrated with the proposed prior model to address the binary linear inverse problem from the Bayesian inference perspective. Numerical examples are presented to demonstrate the effectiveness and robustness of the proposed BG Binary Inversion method. The performance improvement of the proposed method is attributed to the exploitation of the binary prior information of the solution and GAMP technique.

View-based weight network for 3D object recognition

Projective methods generally achieve better results in 3D object recognition in recent years. This may be similar to that human visual 3D shapes rely on various 2D observations which are unconscious on retina. Each projection is treated fairly in existing methods. However, we note that different viewpoint images of the same object have different discriminative features, and only some of images are completely significant. We propose a novel View-based Weight Network (VWN) for 3D object recognition where the different view-based weights are assigned to different projections. The trainable view-level weights are incorporated as a pooling layer of the multi-view residual network. The pooling layer contains 7 sub-layers. Meanwhile, we find a simple unsupervised criterion to evaluate the prediction results before they output. To improve the recognition accuracy, a new multi-channel integrated classifier combining Extreme Learning Machine, KNN, SVM and Random Forest is proposed based on the criterion. The multi-channel classifier can make the accuracy of Top1 close to Top2. Experiments on Princeton ModelNet 3D datasets demonstrate our proposed method outperforms the state-of-the-art approaches significantly in recognition accuracy.

LGS: LOCAL GEOMETRICAL STRUCTURE-BASED INTEREST POINT MATCHING FOR WIDE-BASELINE IMAGERY IN URBAN AREAS

Abstract. Feature matching is a fundamental technical issue in many applications of photogrammetry and remote sensing. Although recently developed local feature detectors and descriptors have contributed to the advancement of point matching, challenges remain with regard to urban area images that are characterized by large discrepancies in viewing angles. In this paper, we define a concept of local geometrical structure (LGS) and propose a novel feature matching method by exploring the LGS of interest points to specifically address difficult situations in matching points on wide-baseline urban area images. In this study, we first detect interest points from images using a popular detector and compute the LGS of each interest point. Then, the interest points are classified into three categories on the basis of LGS. Thereafter, a hierarchical matching framework that is robust to image viewpoint change is proposed to compute correspondences, in which different feature region computation methods, description methods, and matching strategies are designed for various types of interest points according to their LGS properties. Finally, random sample consensus algorithm based on fundamental matrix is applied to eliminate outliers. The proposed method can generate similar feature descriptors for corresponding interest points under large viewpoint variation even in discontinuous areas that benefit from the LGS-based adaptive feature region construction. Experimental results demonstrate that the proposed method provides significant improvements in correct match number and matching precision compared with other traditional matching methods for urban area wide-baseline images.

Adaptive Pixel-wise and Block-wise Stereo Matching in Lighting Condition Changes

The depth information performs a very important role in a production of three-dimensional (3D) video content. One way to acquire this information is a stereo matching method. The stereo matching method searches for correspondences from a stereo image that has two different viewpoints. Subsequently, it estimates the depth information by calculating a disparity value between two corresponding points. Generally, a relatively accurate result is obtained from the correspondence search when the stereo image is captured under uniform illumination and exposure conditions. However, it is difficult to estimate accurate correspondences if each viewpoint image is captured under varying illumination and exposure conditions. In this paper, we analyze conventional pixel-wise and block-wise stereo matching methods that are robust to lighting condition changes. Subsequently, we propose an adaptive pixel-wise and block-wise stereo matching method based on the analysis result.

Joint 3-D Shape Estimation and Landmark Localization From Monocular Cameras of Intelligent Vehicles

3-D reconstruction is at the core for many driving applications of Internet of Intelligent Vehicles. Previous works on reconstruction of a 3-D point shape commonly use a two-step framework. Precisely localizing a series of feature points in an image is performed on the first step. Then the second procedure attempts to fit the 3-D data to the observations to get the real 3-D shape. Such an approach has high time consumption, and easily gets stuck into local minimum. To address this problem, we propose a method to jointly estimate the global 3-D geometric structure of car and localize 2-D landmarks from a single viewpoint image. First, we represent the 3-D shape with a set of predefined shape bases, while parametrizing it by the coefficients of the linear combination of them. Second, we adopt a cascaded regression framework to regress the global shape encoded by the prior bases, by jointly minimizing the appearance and shape fitting differences. The position fitting item can help cope with the description ambiguity of local appearance, and provide more information for 3-D reconstruction. We apply the proposed approach on a multiview car dataset. Experimental results demonstrate favorable improvements on pose estimation and shape prediction, compared with some previous methods.

Super-multiview integral imaging scheme based on sparse camera array and CNN super-resolution.

In this paper, we propose a scheme based on sparse camera array and convolution neural network super-resolution for super-multiview integral imaging. In particular, the proposed scheme is adequate to not only the virtual-world three-dimensional scene with high performance and efficiency, but also the real-world 3D scene with higher availability than the traditional methods. In the proposed scheme, we first adopt the sparse camera array strategy to capture the sparse viewpoint images and use these images to synthesize the low-resolution elemental image array, then the convolution neural network super-resolution scheme is used to restore the high-resolution elemental image array from the low-resolution elemental image array for super-multiview integral image display. Experimental results verify the feasibility of the proposed scheme.

Region Driven Remote Sensing Image Captioning

Abstract Remote sensing Image Captioning is a special case of Image Captioning which solves the difficulties in processing the remote sensing images. Issues may arise due to translation, rotation and viewpoint of images and maintaining semantic consistency in the generated captions. This method of describing a remote sensing scene in the form of sentences plays an important role in a number of fields, such as image retrieval, scene classification and as a vision companion. A Domain-driven approach is developed, in which the domain probabilities are used for captioning the remote sensing images. This approach concentrates on the domain- based information available in the images. A new dataset, called UAVIC dataset is created for images captured using Unmanned Aerial Vehicle (UAV), which covers wide range of land having multiple terrains and gives a better view of the landscapes. The proposed domain driven approach is applied to UCM and UAVIC dataset and the quality of resulting captions are evaluated using BLEU scores.

Removal of Obstacles on Free Viewpoint Image for Tele-operation of Robot

Removal of Obstacles on Free Viewpoint Image for Tele-operation of Robot

自由立体显示中基于深度卷积神经网络的虚拟视点生成方法

自由立体显示中基于深度卷积神经网络的虚拟视点生成方法

As-Aligned-As-Possible Image Stitching Based on Deviation-Corrected Warping With Global Similarity Constraints

Handling local misalignment caused by the local warp remains a common and challenging task for image stitching. Moreover, the stitched image is prone to appearing ghosting due to the variations of the image viewpoint between images. To solve the problem of local misalignment, we propose a projection deviation-corrected local warping method with a global similarity constraint for image stitching. Recent warps prove that the warp of the local mesh guide image effectively improves the accuracy of image alignment. Geometric projection deviation is well used to accurately correct pixel offsets in image warping. To correct pixel offsets, we first remove the outliers from matching points by using the normal distribution model. The retained matches are more precise and can improve the accuracy of image alignment. Next, we use the local warping model combining local homography and global similarity for image warping. To further address the misalignment problem caused by local warping, we describe the local projection deviation of the local warping model by adopting a three-dimensional mesh interpolation model. Finally, the warped images are blended by a linear smoothing model. Experimental results show that our method outperforms the state-of-the-arts in alignment accuracy, and also provides better visual effects on challenging images.

Encouraging Women to Compete: Social Image and Prosocial Incentives

In this paper, we investigate how social image concerns affect women’s decisions to compete. We first propose a theoretical model to show that participating in a competition is costly for women from the social image viewpoint, since such behavior deviates from traditional female gender norms. This results in women’s lower standing in competitive environments, even under affirmative action policies favoring women, when decisions to compete are observed. We posit and theoretically demonstrate that introducing prosocial incentives in a competitive environment can effectively encourage women to compete even under public observability. Second, we conduct a laboratory experiment, randomly manipulate the public observability of decisions to compete, and test our theoretical predictions. The results of the experiment are fairly consistent with the theoretical predictions. We suggest that, when designing policies to promote gender equality in competitive environments, using prosocial incentives through company philanthropy or other social responsibility policies could be promising.