Spherical Panorama Research Articles

Real‐time generation of spherical panoramic video using an omnidirectional multi‐camera system

AbstractThis paper presents a novel method for real‐time generation of seamless spherical panoramic videos from an omnidirectional multi‐camera system (OMS). Firstly, a multi‐view video alignment model called spherical projection constrained thin‐plate spline (SP‐TPS) was established and estimated using an approximately symmetrical seam‐line, maintaining the structure inconsistency around the seam‐line. Then, a look‐up table was designed to support real‐time processing of video re‐projection, video dodging and seam‐line updates. In the table, the overlapping areas in OMS multi‐view videos, the seam‐lines between spherical panoramas and OMS multi‐view videos and the pixel coordinate mapping relationship between spherical panoramas and OMS multi‐view videos were pre‐stored as a whole. Finally, a spherical panoramic video was outputted in real‐time through look‐up table computation under an ordinary GPU processor. The experiments were conducted on multi‐view video taken by “1 + 4” and “1 + 7” OMS, respectively. Experimental results demonstrate that compared with four state‐of‐the‐art methods reported in the literature and two bits of commercial software for video stitching, the proposed method excels in eliminating visual artefacts and demonstrates superior adaptability to scenes with varying depths of field. Assuming that OMS is not movable in the scene, this method can generate seamless spherical panoramic videos with a resolution of 8 K in real time, which is of great value to the surveillance field.

Deep scene synthesis of Atlanta-world interiors from a single omnidirectional image.

We present a new data-driven approach for extracting geometric and structural information from a single spherical panorama of an interior scene, and for using this information to render the scene from novel points of view, enhancing 3D immersion in VR applications. The approach copes with the inherent ambiguities of single-image geometry estimation and novel view synthesis by focusing on the very common case of Atlanta-world interiors, bounded by horizontal floors and ceilings and vertical walls. Based on this prior, we introduce a novel end-to-end deep learning approach to jointly estimate the depth and the underlying room structure of the scene. The prior guides the design of the network and of novel domain-specific loss functions, shifting the major computational load on a training phase that exploits available large-scale synthetic panoramic imagery. An extremely lightweight network uses geometric and structural information to infer novel panoramic views from translated positions at interactive rates, from which perspective views matching head rotations are produced and upsampled to the display size. As a result, our method automatically produces new poses around the original camera at interactive rates, within a working area suitable for producing depth cues for VR applications, especially when using head-mounted displays connected to graphics servers. The extracted floor plan and 3D wall structure can also be used to support room exploration. The experimental results demonstrate that our method provides low-latency performance and improves over current state-of-the-art solutions in prediction accuracy on available commonly used indoor panoramic benchmarks.

ENABLING SCAN-TO-BIM WORKFLOW FOR HERITAGE CONSERVATION AND MANAGEMENT PROCESS

Abstract. The paper stresses the potential of digital technology in the field of Cultural Heritage (CH), focusing on Architectural Heritage (AH) conservation and management process. It presents a workflow based on non-parametric automatic 3D modeling and Industry Foundation Classes (IFC) authoring to combine geometry and semantic information. The goal of such a methodology is to overcome time-consuming manual tasks that still limit the Historical Building Information Modelling (HBIM), which nevertheless proves to be one of the most effective digital tools for the conservation and management of AH in the medium to long term. The proposed workflow consists of five steps: the 3D survey campaign and raw data processing, the semantic annotation of images according to different information layers, the annotation transfer from images to the point cloud representing the studied architecture, the obtained Point Cloud Information Model (PCIM) automatic segmentation and 3D mesh wrapping, and finally the IFC generation for the HBIM representation. This unconventional digital chain was tested to support the knowledge representation of Villa Leonardi, an historical house in the Marche Region (Italy). The final output is a 3D model of the building, automatically processed from digital survey data, and semantically enriched thanks to the material consistency and decay analysis carried out and annotated on spherical panoramas. The effectiveness of this approach was proven in the conservation project for the façades of the house: enriched data on materials and decay supported an action plan of restoration works, which was then recorded in the HBIM model, thus permitting time and cost control.

Free-Viewpoint Navigation of Indoor Scene with 360° Field of View

By providing a 360° field of view, spherical panoramas can convey vivid visual impressions. Thus, they are widely used in virtual reality systems and street view services. However, due to bandwidth or storage limitations, existing systems only provide sparsely captured panoramas and have limited interaction modes. Although there are methods that can synthesize novel views based on captured panoramas, the generated novel views all lie on the lines connecting existing views. Therefore these methods do not support free-viewpoint navigation. In this paper, we propose a new panoramic image-based rendering method for novel view generation. Our method represents each input panorama with a set of spherical superpixels and warps each superpixel individually so the method can deal with the occlusion and disocclusion problem. The warping is dominated by a two-term constraint, which can preserve the shape of the superpixel and ensure it is warped to the correct position determined by the 3D reconstruction of the scene. Our method can generate novel views that are far from input camera positions. Thus, it supports freely exploring the scene with a 360° field of view. We compare our method with three previous methods on datasets captured by ourselves and by others. Experiments show that our method can obtain better results.

Spherical Box-Counting: Combining 360° Panoramas with Fractal Analysis

In this paper, a new box-counting method to achieve a highly specific topological fingerprinting of architecture in relation to the position of the observer and in the context of its surroundings is proposed. Central to this method is the use of 360-degree spherical panoramas as a basis for fractal measurement. Thus, a number of problems of the comparative analysis of the fractal dimension in the field of architecture are explicitly and implicitly addressed, first and foremost being the question of choosing image boundaries while considering adjacent vegetation, urban elements, and other visually present objects for Gestalt analysis of a specific building. Second, the problem of distance and perspective as part of the aesthetic experience based on viewer and object location were taken into account and are addressed. The implications of the use of a spherical perspective as described in this research are also highly relevant for other methods of aesthetic measures in architecture, including those implementing collaborative design processes guided by digital tools and machine learning, among others.

Virtual Walk in the Church of Nativity in Arbanasi

The paper presents the special features of the virtual walk developed in Bulgarian - Hungarian co-operation for the famous Bulgarian church named as Church of Nativity in Arbanasi. The virtual walk is created from high-resolution interactive spherical panorama pictures. The virtual walk is integrated with the Bulgarian Iconographical Digital Library, a rich repository of Bulgarian icons.

ViCT—Virtual Campus Tour Environment with Spherical Panorama: A Preliminary Exploration

ViCT or spelled "VISIT", is the name of a web-based application that is being developed to display various points and locations from the campus environment that can be accessed virtually. ViCT aims to help users, students, and the public explore campus with digital freedom of movement anytime and anywhere, educating users about campus facilities and services with an immersive technology experience that is eco-friendly and fun. The application development concept is unlimited visits and easily accessible via desktop and mobile, saving resources and costs. In this chapter, the focus of the researchers discusses the initial digital mapping of the campus environment, which is the object or study area of the development of virtual campus applications with 360-degree panoramic image techniques. Spherical panorama images are the primary basis of the virtual reality environment in this application. Each panoramic image is processed and merged into a unified whole on a web page. Spherical panorama photos captured using ultra-wide-angle lenses at the front and rear of the camera (dual fisheye) generate a visual field with a horizontal and vertical angle of view close to 360 degrees.

Istoriya odnogo zavodskogo muzeya: ot real'nosti k virtual'nosti

The work is devoted to the issues of information analysis of historical processes related to industrialization in the Tambov region. The beginning of the Tambov car repair workshops (later the car repair plant) turned out to be a landmark event that marked the transition to a new stage of development, changed the economy and to some extent the way of life of the old provincial city. The production activity of the plant is reflected in the museum, created in 1967 in the premises of the factory club. In October 2021, the museum was closed. And the authors of the work, in order to preserve in the fullest possible form the most valuable information contained in the museum being liquidated, using the means of modern computer technologies, transferred the real museum to the virtual world. In the 3D Vista Virtual Tour Pro software environment, a virtual tour of the factory museum has been created, which includes more than 20 spherical panoramas. Thus, one of the most interesting factory museums in the city of Tambov has been preserved in the memory of citizens and all those who are not indifferent to the history of the formation of industry.

THE WHOLE IS OTHER THAN THE SUM OF ITS PARTS: SENSIBILITY ANALYSIS OF 360° URBAN IMAGE SPLITTING

Abstract. 360° imagery has been increasingly used to estimate the subjective qualities of the urban space, such as the feeling of safety or the liveliness of a place. These spherical panoramas offer an immersive view of the urban scene, close to the experience of a pedestrian. In recent years, Deep Learning approaches have been developed for this estimation task, only using flat images because these images are easier to annotate and process with standard CNNs. Thus to qualify the whole urban space, the panoramic images are divided into four flat sub-images that can be processed by the trained neural networks. The sub-images cover the 360° field of view, e.g. front, back, left, and right views. The four scores obtained are averaged to represent the level of the quality at the location of the panorama. However, this split introduces a bias since some elements of the urban space are halved over two images and the global context is lost. Based on the Place Pulse 2.0 dataset, this paper investigates the impact of splitting 360° panoramas on the perceptual scores predicted by neural networks. For each panorama, we predict the score for thirty-six overlapping sub-images. The scores were shown to have high variability and be highly dependent on the direction of the camera for the perspective images. This indicates that four images are not sufficient to capture the complexity of the perceptual qualities of the urban space.

Ray Tracing Acceleration Algorithm Based on FaceMap

Raster method and ray tracing are two important algorithms in computer graphics. The former, raster method, marked by high speed and efficiency, has the disadvantage of an unrealistic rendering effect. By contrast, ray tracing requires considerable computing resources and time despite its advantages of high fidelity and simple structure, thus only suitable for offline rendering. Given such limitations, a ray-tracing acceleration algorithm that combines the raster method and ray tracing is proposed in this study based on FaceMap, a two-dimensional data structure that stores surface distribution under point light or the view of a camera. Firstly, all triangular surfaces of the 3D model are rasterized by linear surface projection to form the surface distribution of a spherical panorama. Secondly, the structure of FaceMap is adopted to store the distribution information of all the surfaces in the scene. Thirdly, the data on the intersecting surfaces in this direction are collected by projecting the ray onto FaceMap in the ray-tracing process, thus reducing the intersection and propagation operations and improving the efficiency. Four object models including chessboard, grass patch, bust, and typewriter are selected for comparative rendering experiments. Our proposed method is used to compare with Rhino and VRay rendering software, respectively. The results show that our proposed method obtained better rendering effects within greatly reduced rendering time (© 2018Optical Society of America).

Digital Tools to Serve Geotourism and Sustainable Development at Psiloritis UNESCO Global Geopark in COVID Times and Beyond

Digital tools that aid geolocation, geointerpretation and geomodelling are increasingly used in the promotion of geoheritage and geoconservation. UNESCO Global Geoparks (UGGps) are complex regions that require a variety of approaches to advance geoconservation and public awareness, holistic heritage management and sustainable development. UGGps need more diversified and applied digital tools to address these subjects. Additional efforts are made through their commitment to achieving sustainable development goals (SDGs) in the changing and challenging world of the COVID-19 pandemic and the exacerbation of climate change. In this study, we present three new digital applications developed for the Psiloritis UGGp in Southern Greece. These digital tools were developed under the implementation of the “Enhancement Plan” of the geopark via the RURITAGE, a project that supports rural regeneration through conservation, with a focus on local heritage. Digital tools developed in the project include an interactive digital map that demonstrates all properties of local heritage, products and services, two story maps focusing on historic churches and monasteries of the Amari district and on the natural and cultural values of Nida plateau, and a business-listing map with the affiliated geopark enterprises. These digital tools combine multiple applications and methods such as Wordpress webpages, web maps, spherical panoramas, multimedia, site interpretation, geolocation and virtual reality to aid the interpretation of natural and cultural heritage, promote important sites, demonstrate overlaps between nature and human society and support local productivity. Digital tools offer online access to interested parties in any area and are also used for in situ information sites. They are user-friendly, device-adjusted and available for sharing on social media and webpages. The applicability and effectiveness of these digital tools are proven to advance geotourism and the SDGs, in line with the provisions of the “World After roadmap” of UGGps. During the COVID-19 pandemic, the “visibility” of the Psiloritis UGGp was doubled via the use of these digital tools, as they have become popular among the general public.

VR360-INSPIRATION: Learning Media on Plant Structure and Function to Improve Student's Analytical Thinking

Analytical thinking is an ability that is really needed in facing the future challenges, including in the industrial era 4.0. The ability to think analytically is important for students to solve problems in a lesson, including the plant structure and function material which is commonly carried out through observation and practicum activities. Limited conditions during the Covid-19 pandemic require mastery of analytical thinking skills in students through innovative learning media. For this reason, the Virtual Reality 360 Interactive Spherical Panorama Slide Presentation (VR360-INSPIRATION) learning media was developed with a 4D research and development model which aims to improve students' analytical thinking skills. Based on the results of the media validation test, the average value was 3.53 in the valid category. The results of the parametric descriptive test and the effectiveness of the media showed that the use of VR360-INSPIRATION media was significantly different from the results of the control group and had high effectiveness in improving students' analytical thinking skills.

Deep3DLayout

Recovering the 3D shape of the bounding permanent surfaces of a room from a single image is a key component of indoor reconstruction pipelines. In this article, we introduce a novel deep learning technique capable to produce, at interactive rates, a tessellated bounding 3D surface from a single 360° image. Differently from prior solutions, we fully address the problem in 3D, significantly expanding the reconstruction space of prior solutions. A graph convolutional network directly infers the room structure as a 3D mesh by progressively deforming a graph-encoded tessellated sphere mapped to the spherical panorama, leveraging perceptual features extracted from the input image. Important 3D properties of indoor environments are exploited in our design. In particular, gravity-aligned features are actively incorporated in the graph in a projection layer that exploits the recent concept of multi head self-attention, and specialized losses guide towards plausible solutions even in presence of massive clutter and occlusions. Extensive experiments demonstrate that our approach outperforms current state of the art methods in terms of accuracy and capability to reconstruct more complex environments.

Spherical panorama compositing through depth estimation

In this paper, we propose to work in the 2.5D space of the scene to facilitate composition of new spherical panoramas. For adding depths to spherical panoramas, we extend an existing method that was designed to estimate relative depths from a single perspective image through user interaction. We analyze the difficulties to interactively provide such depth information for spherical panoramas, through three different types of presentation. Then, we propose a set of basic tools to interactively manage the relative depths of the panoramas in order to obtain a composition in a very simple way. We conclude that the relative depths obtained by the extended depth estimation method are enough for the purpose of compositing new photorealistic panoramas through a few elementary editing tools.

Estimation of forest canopy structure and understory light using spherical panorama images from smartphone photography

Abstract Accurate estimates of forest canopy structure are central for a wide range of ecological studies. Hemispherical photography (HP) is a popular tool to estimate canopy attributes. However, traditional HP methods require expensive equipment, are sensitive to exposure settings, and produce limited resolution which dramatically affects the accuracy of gap fraction estimates. As an alternative, hemispherical images can be extracted from spherical panoramas produced by many smartphone camera applications. I compared hemispherical photos captured with a digital single lens reflex camera and 180° lens to those extracted from smartphone spherical panoramas (SSP). The SSP HP method leverages built-in features of current generation smartphones to produce sharper images of higher resolution, resulting in more definition of fine canopy structure. Canopy openness and global site factor from SSP HP are highly correlated with traditional methods (R2 > 0.9), while leaf area index estimates are lower, especially in more closed canopies where traditional methods fail to capture fine gaps.

UniFuse: Unidirectional Fusion for 360° Panorama Depth Estimation

Learning depth from spherical panoramas is becoming a popular research topic because a panorama has a full field-of-view of the environment and provides a relatively complete description of a scene. However, applying well-studied CNNs for perspective images to the standard representation of spherical panoramas, i.e., the equirectangular projection, is suboptimal, as it becomes distorted towards the poles. Another representation is the cubemap projection, which is distortion-free but discontinued on edges and limited in the field-of-view. This letter introduces a new framework to fuse features from the two projections, unidirectionally feeding the cubemap features to the equirectangular features only at the decoding stage. Unlike the recent bidirectional fusion approach operating at both the encoding and decoding stages, our fusion scheme is much more efficient. Besides, we also designed a more effective fusion module for our fusion scheme. Experiments verify the effectiveness of our proposed fusion strategy and module, and our model achieves state-of-the-art performance on four popular datasets. Additional experiments show that our model also has the advantages of model complexity and generalization capability.

Single-shot cuboids: Geodesics-based end-to-end Manhattan aligned layout estimation from spherical panoramas

It has been shown that global scene understanding tasks like layout estimation can benefit from wider field of views, and specifically spherical panoramas. While much progress has been made recently, all previous approaches rely on intermediate representations and postprocessing to produce Manhattan-aligned estimates. In this work we show how to estimate full room layouts in a single-shot, eliminating the need for postprocessing. Our work is the first to directly infer Manhattan-aligned outputs. To achieve this, our data-driven model exploits direct coordinate regression and is supervised end-to-end. As a result, we can explicitly add quasi-Manhattan constraints, which set the necessary conditions for a homography-based Manhattan alignment module. Finally, we introduce the geodesic heatmaps and loss and a boundary-aware center of mass calculation that facilitate higher quality keypoint estimation in the spherical domain. Our models and code are publicly available at https://github.com/VCL3D/SingleShotCuboids.

360-Degree VR Video Watermarking Based on Spherical Wavelet Transform

Similar to conventional video, the increasingly popular 360 virtual reality (VR) video requires copyright protection mechanisms. The classic approach for copyright protection is the introduction of a digital watermark into the video sequence. Due to the nature of spherical panorama, traditional watermarking schemes that are dedicated to planar media cannot work efficiently for 360 VR video. In this article, we propose a spherical wavelet watermarking scheme to accommodate 360 VR video. With our scheme, the watermark is first embedded into the spherical wavelet transform domain of the 360 VR video. The spherical geometry of the 360 VR video is used as the host space for the watermark so that the proposed watermarking scheme is compatible with the multiple projection formats of 360 VR video. Second, the just noticeable difference model, suitable for head-mounted displays (HMDs), is used to control the imperceptibility of the watermark on the viewport. Third, besides detecting the watermark from the spherical projection, the proposed watermarking scheme also supports detecting watermarks robustly from the viewport projection. The watermark in the spherical domain can protect not only the 360 VR video but also its corresponding viewports. The experimental results show that the embedded watermarks are reliably extracted both from the spherical and the viewport projections of the 360 VR video, and the robustness of the proposed scheme to various copyright attacks is significantly better than that of the competing planar-domain approaches when detecting the watermark from viewport projection.

Elaboration of New Viewing Modes in CATIA CAD for Lighting Simulation Purpose

During integration of lighting simulation systems into CATIA CAD often are needed additional view modes initially absent in this CAD. This article describes adding such view modes directly in CATIA workspace window using CATIA’s native CAA RADE programming environment. Adding these new view modes can significantly simplify user’s preparation of data for lighting simulation. Here the following additional view modes are mentioned: geometry tessellation to triangular meshes (in the wireframe mode), heat map dis-play of the optical values distribution over the scene, and spherical panorama preview. In the paper the problems which may occur while trying to redraw the additional data in real time are described and possible ways of solving or by-passing them are proposed. Using the new display modes provides a possibility of tuning some scene parameters and viewing preliminary results immediately while rendering of such scenes may take several hours. Such additional functionality can significantly increase the modeling process.

An algorithm for the restoration of blurred image obtained with a rotating camera tilted to the horizon

This work is a continuation of the authors’ previous publications, in which the restoration of images obtained with a horizontally rotating camera was considered. In this paper, a mathematical model is constructed for reconstructing blurred images obtained with a camera rotating in the horizontal plane and having its optical axis tilted at an angle to the horizon. The method of image restoration involves constructing a strip of spherical panorama from the original images. Results of numerical experiments that confirm the good quality of the presented image recovery method and the high performance of the developed algorithm are presented.