Adjacent Viewpoints Research Articles

Visual angle enlargement method based on effective reconstruction area

The traditional integral imaging system has a small viewing angle, mainly due to the existence of interference pixels in the generated element image. At the same time, in the 3D reconstruction scene, the content that is not in the effective reconstruction area will also cause image crosstalk between adjacent viewpoints, which affects the display angle. In this paper, a viewing angle enlargement method based on an effective reconstruction area is proposed to establish the correspondence between the point pixels in the element image and the micro-lens in the micro-lens array, and the pixel information in the pixel recombination area is derived and updated. The effective reconstruction area is used for constraint, so as to enhance the effective use of the element image information. The experimental results show that the element image with the pixel update has a larger viewing angle and clearer reconstruction content.

Iterative extrinsic calibration using virtual viewpoint for 3D reconstruction

We propose a camera calibration to generate a high-quality and photorealistic 3D (dimension) volumetric model using several low-cost RGB-D (depth) cameras located in a space. We show an efficient workflow to register a model and propose iterative calibration to construct it. Using multiple frames, calibration in the vertical direction between the upper and lower cameras is performed. After selecting any four pairs, the calibration is performed while rotating with the vertical calibration results from other adjacent viewpoints. After performing calibration between each camera pair, the calibration is repeated by generating a virtual viewpoint between each camera pair. The error function between 3D coordinates of feature points acquired from the RGB image is obtained, and an attempt is made to minimize this. When the error converges below a threshold value by optimizing the error function, calibration ends, and the final extrinsic parameter is obtained. After performing 3D reconstruction using the proposed calibration, a 3D point cloud is produced. Finally, a simple and efficient refinement is proposed to improve the 3D point cloud quality. We show the advantage of the proposed technique by quantitatively comparing the calibration results using two ground truth data and 3D reconstruction results in the experimental results.

Adjustable and continuous eyebox replication for a holographic Maxwellian near-eye display.

A Maxwellian display presents always-focused images to the viewer, alleviating the vergence-accommodation conflict (VAC) in near-eye displays (NEDs). Recently, many methods of improving its limited eyebox have been proposed, among which viewpoint replication has attracted a lot of attention. However, double-image, blind-area, and image-shift effects always happen in typical eyebox-replication Maxwellian NEDs when the eye moves between the replicated viewpoints, which prevents these NEDs from being applied more widely. In this Letter, we propose a method for designing a holographic Maxwellian NED system with continuous eyebox replication as well as flexible interval adjustment by changing the projection angles of the reconstructed images. Thus, holograms corresponding to the positions of different viewpoints are calculated to match the interval of the replicated viewpoints with the human pupil diameter, making it possible to eliminate or alleviate double-image or blind-area effects. Also, seamless viewpoint conversion in the eyebox is achieved by aligning the images of adjacent viewpoints on the retina via hologram pre-processing independently. These effects are verified successfully in optical experiments and have the potential to be applied in near-eye three-dimensional displays without VAC.

Multi-Robot Hybrid Coverage Path Planning for 3D Reconstruction of Large Structures

Coverage Path Planning (CPP) is an essential capability for autonomous robots operating in various critical applications such as fire fighting, and inspection. Performing autonomous coverage using a single robot system consumes time and energy. In particular, 3D large structures might contain some complex and occluded areas that shall be scanned rapidly in certain application domains. In this paper, a new Hybrid Coverage Path Planning (HCPP) approach is proposed to explore and cover unknown 3D large structures using a decentralized multi-robot system. The HCPP approach combines a guided Next Best View (NBV) approach with a developed Long Short Term Memory (LSTM) waypoint prediction approach to decrease the CPP exploration time at each iteration and simultaneously achieve high coverage. The hybrid approach is the new ML paradigm which fosters intelligence by balancing between data efficiency and generality allowing the exchange of some CPP parts with a learned model. The HCPP uses a stateful LSTM network architecture which is trained based on collected paths that cover different 3D structures to predict the next viewpoint. This architecture captures the dynamic dependencies of adjacent viewpoints in the long term sequences like the coverage paths. The HCPP switches between these methods triggered by either the number of iterations or an entropy threshold. In the decentralized multi-robot system, the proposed HCPP is embedded in each robot where each one of them shares its global 3D map ensuring robustness. The results performed in a realistic Gazebo robotic simulator confirmed the advantage of the proposed HCPP approach by achieving high coverage on different 3D unknown structures in a shorter time compared to conventional NBV.

P‐4.14: Viewing zones of Lenticular‐Auto‐stereoscopic 3D system

In this paper, the geometric model of the viewing zone is established and the factors that influence the distribution of the viewing zone are analyzed. For the first time, the impact of input image of display panel is considered. Through simulation, not only the best viewing distance and the distance between adjacent viewpoints can be obtained, but also the change of the viewing zone with the input image can be obtained. The geometric model makes the calculation of the view zone and the design layout more accurate and simple.

Best Viewpoints for External Robots or Sensors Assisting Other Robots

This work creates a model of the value of different external viewpoints of a robot performing tasks. The current state of the practice is to use a teleoperated assistant robot to provide a view of a task being performed by a primary robot; however, the choice of viewpoints is ad hoc and does not always lead to improved performance. This research applies a psychomotor approach to develop a model of the relative quality of external viewpoints using Gibsonian affordances. In this approach, viewpoints for the affordances are rated based on the psychomotor behavior of human operators and clustered into manifolds of viewpoints with the equivalent value. The value of 30 viewpoints is quantified in a study with 31 expert robot operators for four affordances (reachability, passability, manipulability, and traversability) using a computer-based simulator of two robots. The adjacent viewpoints with similar values are clustered into ranked manifolds using agglomerative hierarchical clustering. The results show the validity of the affordance-based approach by confirming that there are manifolds of statistically significantly different viewpoint values, viewpoint values are statistically significantly dependent on the affordances, and viewpoint values are independent of a robot. Furthermore, the best manifold for each affordance provides a statistically significant improvement with a large Cohen's d effect size (1.1-2.3) in the performance (improving time by 14%-59% and reducing errors by 87%-100%) and improvement in the performance variation over the worst manifold. This model will enable autonomous selection of the best possible viewpoint and path planning for the assistant robot.

A ghostfree contrast enhancement method for multiview images without depth information

High dynamic range (HDR) images greatly improve visual content quality, but pose challenges in processing, acquisition, and display. Images captured in real-world scenarios with multiple nonlinear cameras, extremely short unknown exposure time, and a shared light source present the additional challenges of incremental baseline and angle deviation amongst the cameras. The disparity maps in such conditions are not reliable; therefore, we propose a method that relies on the accurate detection and matching of feature points across adjacent viewpoints. We determine the exposure gain among the matched feature points in the involved views and design an image restoration method to restore multiview low dynamic range (MVLDR) images for each viewpoint. Finally, the fusion of these restored MVLDR images produces high-quality images for each viewpoint without capturing a series of bracketed exposure. Extensive experiments are conducted in controlled and uncontrolled conditions, and results prove that the proposed method competes for the state-of-the-arts.

A Smooth Transition Algorithm for Adjacent Panoramic Viewpoints Using Matched Delaunay Triangular Patches

The unnatural panoramic image transition between two adjacent viewpoints reduces the immersion and interactive experiences of 360° panoramic walkthrough systems. In this paper, a dynamic panoramic image rendering and smooth transition algorithm for adjacent viewpoints is proposed. First, the feature points of adjacent view images are extracted, a robust matching algorithm is used to establish adjacent point pairs, and the matching triangles are formed by using the homonymous points. Then, a dynamic transition model is formed by the simultaneous linear transitions of shape and texture for each control triangle. Finally, the smooth transition between adjacent viewpoints is implemented by overlaying the dynamic transition model with the 360° panoramic walkthrough scene. Experimental results show that this method has obvious advantages in visual representation with distinct visual movement. It can realize the smooth transition between two indoor panoramic stations with arbitrary station spacing, and its execution efficiency is up to 50 frames per second. It effectively enhances the interactivity and immersion of 360° panoramic walkthrough systems.

Fast 3D-HEVC inter mode decision algorithm based on the texture correlation of viewpoints

To reduce the computational complexity of 3D-HEVC, a fast inter mode decision algorithm based on texture correlation of adjacent viewpoints is proposed (FMD_inter). By studying the statistical probability of inter prediction mode distributions in 3D-HEVC coding scheme, the mode of dependent view is highly correlated with the base one, which enables the early mode decision. Moreover, by analyzing the relationship between video texture and the inter prediction mode, selectively skip checking symmetric motion partition (SMP) mode and asymmetric motion partition (AMP) mode according to the texture features of coding unit is reasonable. The proposed algorithm is able to achieve a reduction of computational complexity by 18.71% on average, while maintaining the coding efficiency. The subjective quality of the synthesized views also shows the effectiveness of the algorithm.

21‐1: Reducing Image Quality Variation with Motion Parallax for Glassless 3D Screens using Linear Blending Technology

The linear blending of viewing zones is effective for reducing the number of projectors of a multi‐view 3D screen. However, the image quality of intermediate viewpoint degrades because the adjacent viewpoint's images overlap. We propose a method to generate the viewpoint images for linear blending by adding a phase‐based disparity. By image quality evaluation, we show that our method reduces image quality variation.

Human motion analysis from UAV video

PurposeThe purpose of this paper is to present a preliminary solution to address the problem of estimating human pose and trajectory by an aerial robot with a monocular camera in near real time.Design/methodology/approachThe distinguishing feature of the solution is a dynamic classifier selection architecture. Each video frame is corrected for perspective using projective transformation. Then, a silhouette is extracted as a Histogram of Oriented Gradients (HOG). The HOG is then classified using a dynamic classifier. A class is defined as a pose-viewpoint pair, and a total of 64 classes are defined to represent a forward walking and turning gait sequence. The dynamic classifier consists of a Support Vector Machine (SVM) classifier C64 that recognizes all 64 classes, and 64 SVM classifiers that recognize four classes each – these four classes are chosen based on the temporal relationship between them, dictated by the gait sequence.FindingsThe solution provides three main advantages: first, classification is efficient due to dynamic selection (4-class vs 64-class classification). Second, classification errors are confined to neighbors of the true viewpoints. This means a wrongly estimated viewpoint is at most an adjacent viewpoint of the true viewpoint, enabling fast recovery from incorrect estimations. Third, the robust temporal relationship between poses is used to resolve the left-right ambiguities of human silhouettes.Originality/valueExperiments conducted on both fronto-parallel videos and aerial videos confirm that the solution can achieve accurate pose and trajectory estimation for these different kinds of videos. For example, the “walking on an 8-shaped path” data set (1,652 frames) can achieve the following estimation accuracies: 85 percent for viewpoints and 98.14 percent for poses.

Displaying Real-World Light Fields With Stacked Multiplicative Layers: Requirement and Data Conversion for Input Multiview Images

To provide realistic 3-D perception to human observers, 3-D displays have evolved to present not only a pair of stereo images but also many images to different viewing directions. A light-field display, which consists of a few light-attenuating pixelized layers (e.g., LCD panels) stacked in front of a backlight, has attracted attention because of its potential to simultaneously support many viewing directions with high quality. The transmittances of the layers are determined from a set of multiview images or a light field that is given to represent observations expected from many viewing directions. However, the relation between the configuration of the given light field and the quality of displayed images has not sufficiently been discussed in previous works. In this study, in our aim to display real-world objects with high quality, we address the requirement for the target light field given as the input. We deeply analyze several factors that associate the configuration of the target light field with the quality of the displayed images, and we derive a quantitative requirement on the configuration: The disparities among the adjacent viewpoints should be limited to 0–1 pixels. To meet this strict requirement with real-world objects, we propose using a multiview camera and image-based rendering, where we can generate virtual light fields with arbitrary configurations and densities. Our theory and method are verified by experiments using a computer-simulated display.

Multi-view transition HMMs based view-invariant human action recognition method

View-invariant human action recognition is a challenging research topic in computer vision. Hidden Markov Models(HMM) and their extensions have been widely used for view-invariant action recognition. However those methods are usually according to a large parameter space, requiring amounts of training data and with low classification accuracies for real application. A novel graphical structure based on HMM with multi-view transition is proposed to model the human action with viewpoint changing. The model consists of multiple sub action models, which correspond to the traditional HMM utilized to model the human action in a particular rotation viewpoint space. In the training process, the novel model can be built by connecting the sub action models between adjacent viewpoint spaces. In the recognition process, action with unknown viewpoint is recognized by using improved forward algorithm. The proposed model can not only simplify the model training process by decomposing the parameter space into multiple sub-spaces, but also improve the performance the algorithm by constraining the possible viewpoint changing. Experiment results on IXMAS dataset demonstrated that the proposed model obtains better performance than other recent view-invariant action recognition method.