Sequence Of Panoramic Images Research Articles

Large-Scale Indoor Visual-Geometric Multimodal Dataset and Benchmark for Novel View Synthesis.

The accurate reconstruction of indoor environments is crucial for applications in augmented reality, virtual reality, and robotics. However, existing indoor datasets are often limited in scale, lack ground truth point clouds, and provide insufficient viewpoints, which impedes the development of robust novel view synthesis (NVS) techniques. To address these limitations, we introduce a new large-scale indoor dataset that features diverse and challenging scenes, including basements and long corridors. This dataset offers panoramic image sequences for comprehensive coverage, high-resolution point clouds, meshes, and textures as ground truth, and a novel benchmark specifically designed to evaluate NVS algorithms in complex indoor environments. Our dataset and benchmark aim to advance indoor scene reconstruction and facilitate the creation of more effective NVS solutions for real-world applications.

REGISTRATION OF MMS LiDAR POINTS AND PANORAMIC IMAGE SEQUENCE USING RELATIVE ORIENTATION MODEL

Abstract. We propose using the relative orientation model (ROM) of panoramic to register the MMS LiDAR points and panoramic image sequence, which has the wide applicability. The feature points, extracted and matched from panoramic image pairs, are used to solve the relative position and attitude parameters in the ROM, then, combining the absolute position and attitude parameters of the initial panoramic image, the MMS LiDAR points and panoramic image sequence are registered. First, we propose the position/attitude ROM (PA-ROM) and attitude ROM (A-ROM) of panoramic images respectively, which are apply to the position/attitude parameters both unknown and only the attitude parameters unknown. Second, we automatically extract and match feature points from panoramic image pairs using the SURF algorithm, as these mismatching points will affect the registration accuracy, the RANSAC algorithm and ROM were used to choose the best matching points automatically. Finally, we select the feature points manually from MMS LiDAR points and panoramic image sequence as the checkpoints, and then compare the registration accuracy of continuous/discontinuous panoramic image pairs. The results show that MMS LiDAR points and panoramic image sequence are registered accurately based on ROM (7.36 and 3.75 pixels in dataset I and II), what's more, our registration method just tackle the image pairs (uninvolved LiDAR points), so it is suitable for more road scenes.

Image-Based Camera Localization Algorithm for Smartphone Cameras Based on Reference Objects

Navigating an individual in both outdoor and indoor environments would provide great facilities for the community. Image-based approaches are more affordable and feasible in comparison to the high cost of sensor devices and unavailability of guiding by GPS in indoor systems. Thus, we develop our system using cameras owing to their easy availability and accessibility. Primarily, detecting the location of personnel is needed to orientate a human. However, determining camera localization is a challenging task on the computer vision field. Therefore, in this study, a novel approach is proposed to address the problem using panoramic image sequences. We apply the scale invariant future transform for detecting corresponding points, and based on these points, a reference object is considered. The Levenberg–Marquardt optimization method is utilized to minimize reprojection errors for a given pair of images. As a better alternative to traditional methods, which apply the known surface of a reference object, our formulation rests on the scale ambiguity in different coordinate systems. Experiments are performed on Herz-Jesus-P8, Fountain-P11, and Fountain-P25 datasets to evaluate our dataset and compare our proposal with respect to prior works. We believe our proposal comparatively outperforms previous works indicating lower error rate in wide baselines between camera pairs.

Panoramic SLAM from a multiple fisheye camera rig

This paper presents a feature-based simultaneous localization and mapping (SLAM) system for panoramic image sequences obtained from a multiple fisheye camera rig in a wide baseline mobile mapping system (MMS). First, the developed fisheye camera calibration method combines an equidistance projection model and trigonometric polynomial to achieve high-accuracy calibration from the fisheye camera to an equivalent ideal frame camera, which warrants an accurate transform from the fisheye images to a corresponding panoramic image. Second, we developed a panoramic camera model, corresponding bundle adjustment with a specific back propagation error function, and linear pose initialization algorithm. Third, the implemented feature-based SLAM pipeline consists of several specific strategies and algorithms in initialization, feature matching, frame tracking, and loop closing to overcome the difficulties in tracking wide baseline panoramic image sequences. We conducted experiments on large-scale MMS datasets of more than 15 km trajectories and 14,000 panoramic images as well as small-scale public video datasets. Our results show that the developed panoramic SLAM system PAN-SLAM can achieve fully-automatic camera localization and sparse map reconstruction in both small-scale indoor and large-scale outdoor environments including challenging scenes (e.g., dark tunnel) without the aid of any other sensors. The localization accuracy, which was measured by the absolute trajectory error (ATE), resembled the high-accuracy GNSS/INS reference of 0.1 m. PAN-SLAM also outperformed several feature-based fisheye and monocular SLAM systems with incomparable robustness in various environments. The system could be considered as a robust complementary solution and an alternative to expensive commercial navigation systems, especially in urban environments where signal obstruction and multipath interference are common. Source code and demo are available at http://study.rsgis.whu.edu.cn/pages/download/.

Semiautomatically Register MMS LiDAR Points and Panoramic Image Sequence Using Road Lamp and Lane

We propose using the feature points of road lamp and lane to register mobile mapping system (MMS) LiDAR points and panoramic image sequence. Road lamp and lane are the common sobjects on roads; the spatial distributions are regular, and thus our registration method has wide applicability and high precision. First, the road lamp and lane were extracted from the LiDAR points by horizontal grid and reflectance intensity and then by optimizing the endpoints as the feature points of road lamp and lane. Second, the feature points were projected onto the panoramic image by initial parameters and then by extracting corresponding feature points near the projection location. Third, the direct linear transformation method was used to solve the registration model and eliminate mismatching feature points. In the experiments, we compare the accuracy of our registration method with other registration methods by a sequence of panoramic images. The results show that our registration method is effective; the registration accuracy of our method is less than 10 pixels and averaged 5.84 pixels in all 31 panoramic images (4000 × 8000 pixels), which is much less than that of the 56.24 pixels obtained by the original registration method.

Interactive 3D building modeling method using panoramic image sequences and digital map

This paper proposes a method of generating 3D building models with precise geospatial information and a photograph-based facade appearance from panoramic image sequences and digital maps. 3D building modeling research is actively being conducted in areas such as geographic information systems, virtual reality, and augmented reality. However, the generation of realistic 3D models from a ground-level viewpoint is still extremely costly in terms of labor of modeling experts, and collection of data. We have developed a method for 3D building modeling with high-resolution photograph-based appearance information using panoramic images captured at ground level with a mobile mapping system, and geospatial information obtained from a digital map. The proposed method includes 1) pre-processing for tilt correction and base 3D model generation, 2) geo-registration of panoramic images with minimal user input, and 3) building height and shape estimation. This paper presents the proposed method and the quantitative performance measure obtained from a developed test modeling system. In addition, modeling results from an experimental dataset are also presented.

Automatic registration of panoramic image sequence and mobile laser scanning data using semantic features

Inaccurate exterior orientation parameters (EoPs) between sensors obtained by pre-calibration leads to failure of registration between panoramic image sequence and mobile laser scanning data. To address this challenge, this paper proposes an automatic registration method based on semantic features extracted from panoramic images and point clouds. Firstly, accurate rotation parameters between the panoramic camera and the laser scanner are estimated using GPS and IMU aided structure from motion (SfM). The initial EoPs of panoramic images are obtained at the same time. Secondly, vehicles in panoramic images are extracted by the Faster-RCNN as candidate primitives to be matched with potential corresponding primitives in point clouds according to the initial EoPs. Finally, translation between the panoramic camera and the laser scanner is refined by maximizing the overlapping area of corresponding primitive pairs based on the Particle Swarm Optimization (PSO), resulting in a finer registration between panoramic image sequences and point clouds. Two challenging urban scenes were experimented to assess the proposed method, and the final registration errors of these two scenes were both less than three pixels, which demonstrates a high level of automation, robustness and accuracy.

Geo-registration of wide-baseline panoramic image sequences using a digital map reference

This paper describes a method of geo-registering a sequence of panoramic images to a digital map by matching pixel information from the images with information on the building footprint contained in a digital map. Recently, images captured at the ground level using a Mobile Mapping System (MMS), such as the panoramic images displayed by Google Street View, have been considered as a valuable resource for three-dimensional (3D) building modeling. However, the wide intervals between these panoramic images, as well as locational and directional error from the related sensors, make it difficult to analyze the image data. This paper demonstrates a formulation method for connecting pixels in panoramic images with information on footprint vertices and building lines contained in a digital map. To allow both pixel and footprint information consistent in 3D space, each panoramic image is tilt-corrected in pre-processing to upright the image using the estimated pitch and roll of a vehicle and removing the pitch and roll effects from the panoramic image pixels. Through the proposed formulation, a single panoramic image can be easily geo-registered with simple user-provided constraints, and adjacent sequential images can then be automatically geo-registered using point feature matching. Experimental results showed a significant reduction in the locational and directional error of sequential panoramic images, and the proposed vanishing point (VP) based validation process was found to successfully detect failure cases.

Particle filtering methods for georeferencing panoramic image sequence in complex urban scenes

Georeferencing image sequences is critical for mobile mapping systems. Traditional methods such as bundle adjustment need adequate and well-distributed ground control points (GCP) when accurate GPS data are not available in complex urban scenes. For applications of large areas, automatic extraction of GCPs by matching vehicle-born image sequences with geo-referenced ortho-images will be a better choice than intensive GCP collection with field surveying. However, such image matching generated GCP’s are highly noisy, especially in complex urban street environments due to shadows, occlusions and moving objects in the ortho images. This study presents a probabilistic solution that integrates matching and localization under one framework. First, a probabilistic and global localization model is formulated based on the Bayes’ rules and Markov chain. Unlike many conventional methods, our model can accommodate non-Gaussian observation. In the next step, a particle filtering method is applied to determine this model under highly noisy GCP’s. Owing to the multiple hypotheses tracking represented by diverse particles, the method can balance the strength of geometric and radiometric constraints, i.e., drifted motion models and noisy GCP’s, and guarantee an approximately optimal trajectory. Carried out tests are with thousands of mobile panoramic images and aerial ortho-images. Comparing with the conventional extended Kalman filtering and a global registration method, the proposed approach can succeed even under more than 80% gross errors in GCP’s and reach a good accuracy equivalent to the traditional bundle adjustment with dense and precise control.

Complex monitoring of the state of sea water basins by optical methods. Part 2. Sea surface pollution registration

Optical instruments and methods for a rapid and complex monitoring of coastal waters are analyzed. Usability of polarization panoramic shore-based optical systems for mapping and estimating the dynamic parameters of sea surface pollution with organic films is shown. A possibility of estimating the near-surface wind velocity by processing a temporal sequence of panoramic images is considered. In addition, this video system can be used for real-time direction of research vessels to hydrologic objects (vortices, internal waves, sea surface pollution, etc.) in coastal zones via mobile Internet.

Local and global responses of insect motion detectors to the spatial structure of natural scenes

As a consequence of the non-linear correlation mechanism underlying motion detection, the variability in local pattern structure and contrast inherent within natural scenes profoundly influences local motion responses. To accurately interpret optic flow induced by self-motion, neurons in many dipteran flies smooth this "pattern noise" by wide-field spatial integration. We investigated the role that size and shape of the receptive field plays in smoothing out pattern noise in two unusual hoverfly optic flow neurons: one (HSN) with an exceptionally small receptive field and one (HSNE) with a larger receptive field. We compared the local and global responses to a sequence of panoramic natural images in these two neurons with a parsimonious model for elementary motion detection weighted for their spatial receptive fields. Combined with manipulation of size and contrast of the stimulus images, this allowed us to separate spatial integration properties arising from the receptive field, from other local and global non-linearities, such as motion adaptation and dendritic gain control. We show that receptive field properties alone are poor predictors of the response to natural scenes. If anything, additional non-linearity enhances the pattern dependence of HSN's response, particularly to vertically elongated features, suggesting that it may serve a role in forward fixation during hovering.

MATCHING SEGMENTATION HIERARCHIES

When matching regions from "similar" images, one typically has the problem of missing counterparts due to local or even global variations of segmentation fineness. Matching segmentation hierarchies, however, not only increases the chances of finding counterparts, but also allows us to exploit the manifold constraints coming from the topological relations between any two regions in a hierarchy. To define the topological relations we represent a plane image ℐ by a plane attributed graph G and derive a finite topology [Formula: see text] from G. In particular, segmenting ℐ corresponds to taking a topological minor of G which, in turn, is equivalent to coarsening [Formula: see text]. Moreover, each finite topology involved is a coarsening of the standard topology on ℝ2. Then, we construct a weighted association graph GA, the nodes of which represent potential matches and the edges of which indicate topological consistency with respect to [Formula: see text]. Specifically, a maximal weight clique of GAcorresponds to a topologically consistent mapping with maximal total similarity. To find "heavy" cliques, we extend a greedy pivoting-based heuristic to the weighted case. Experiments on pairs of stereo images, on a video sequence of a cluttered outdoor scene, and on a sequence of panoramic images demonstrate the effectiveness of our method.