Panoramic Scene Research Articles

Panoramic single-pixel imaging with megapixel resolution based on rotational subdivision.

Single-pixel imaging (SPI) using a single-pixel detector is an unconventional imaging method that has great application prospects in many fields to realize high-performance imaging. In particular, the recently proposed catadioptric panoramic ghost imaging (CPGI) extends the application potential of SPI to high-performance imaging at a wide field of view (FOV) with recent growing demands. However, the resolution of CPGI is limited by the hardware parameters of the digital micromirror device (DMD), which cannot meet ultrahigh-resolution panoramic imaging needs that require detailed information. Therefore, to overcome the resolution limitation of CPGI, we propose a panoramic SPI based on rotational subdivision (RSPSI). The key of RSPSI is to obtain the entire panoramic scene by the rotation-scanning of a rotating mirror tilted 45°, so that one single pattern that only covers one sub-FOV with a small FOV can complete an uninterrupted modulation on the entire panoramic FOV during a once-through pattern projection. Then, based on temporal resolution subdivision, the image sequence of sub-FOVs subdivided from the entire panoramic FOV can be reconstructed with pixel-level or even subpixel-level horizontal shifting adjacently. Experimental results using a proof-of-concept setup show that the panoramic image can be obtained with 10428 × 543 of 5,662,404 pixels, which is more than 9.6 times higher than the resolution limit of the CPGI using the same DMD. To the best of our knowledge, the proposed RSPSI is the first to achieve a megapixel resolution via SPI, which can provide potential applications in fields requiring imaging with ultrahigh-resolution and wide FOV.

Multi-Object Tracking Model Based on Detection Tracking Paradigm in Panoramic Scenes

Multi-Object Tracking (MOT) technology is dedicated to continuously tracking multiple targets of interest in a sequence of images and accurately identifying their specific positions at different times. This technology is crucial in key application areas such as autonomous driving and security surveillance. However, the application process often requires the coordination of cameras from multiple angles for tracking. Directly studying Multi-Object Tracking algorithms in panoramic scenes is an effective way to address this issue. The uniqueness of panoramic scenes causes target position changes at the boundaries and tracking difficulties due to continuous changes in target scales. To ensure the accuracy of target tracking, this study explores a detection-based tracking method using the newly improved YOLOx detector and the adjusted DeepSORT algorithm. Firstly, YOLOx_s was chosen as the detector because its simple network structure ensures a fast computational speed. During the feature extraction stage, we used the Polarized Self-Attention (PSA) mechanism to capture more feature information, thereby improving the tracking performance on small-scale targets. Secondly, the tracker was improved by adding a camera motion compensation module before predicting the target’s position to mitigate the impact of camera shake on tracking. Finally, to address the difficulty of continuously tracking targets in specific areas of panoramic scenes, this study proposes specific tracking strategies. These strategies effectively resolve the problem of tracking failure caused by target position changes at the boundaries. Experimental results show that improved algorithms have a superior performance on multiple evaluation metrics compared to other algorithms in the field. Compared to the original algorithm, the improved algorithm exhibits a 6% increase in the quantitative metric MOTA, a 7% increase in IDF1, and a 40% decrease in IDSWs, demonstrating its leading performance.

Motion-Prediction-Based Wireless Scheduling for Interactive Panoramic Scene Delivery

Motion-Prediction-Based Wireless Scheduling for Interactive Panoramic Scene Delivery

Dream360: Diverse and Immersive Outdoor Virtual Scene Creation via Transformer-Based 360° Image Outpainting.

360° images, with a field-of-view (FoV) of $180^{\circ}\times 360^{\circ}$, provide immersive and realistic environments for emerging virtual reality (VR) applications, such as virtual tourism, where users desire to create diverse panoramic scenes from a narrow FoV photo they take from a viewpoint via portable devices. It thus brings us to a technical challenge: 'How to allow the users to freely create diverse and immersive virtual scenes from a narrow FoV image with a specified viewport?' To this end, we propose a transformer-based 360° image outpainting framework called Dream360, which can generate diverse, high-fidelity, and high-resolution panoramas from user-selected viewports, considering the spherical properties of 360° images. Compared with existing methods, e.g., [3], which primarily focus on inputs with rectangular masks and central locations while overlooking the spherical property of 360° images, our Dream360 offers higher outpainting flexibility and fidelity based on the spherical representation. Dream360 comprises two key learning stages: (I) codebook-based panorama outpainting via Spherical-VQGAN (S-VQGAN), and (II) frequency-aware refinement with a novel frequency-aware consistency loss. Specifically, S-VQGAN learns a sphere-specific codebook from spherical harmonic (SH) values, providing a better representation of spherical data distribution for scene modeling. The frequency-aware refinement matches the resolution and further improves the semantic consistency and visual fidelity of the generated results. Our Dream360 achieves significantly lower Frechet Inception Distance (FID) scores and better visual fidelity than existing methods. We also conducted a user study involving 15 participants to interactively evaluate the quality of the generated results in VR, demonstrating the flexibility and superiority of our Dream360 framework.

Uncovering the connection between ceiling height and emotional reactions in art galleries with editable 360-degree VR panoramic scenes.

This study investigates the relationship between ceiling height and emotional responses in art galleries, using editable 360-degree VR panoramic scenes. Prior research has explored the influence of spatial dimensions on general emotions, but the specific impact of ceiling height in art gallery settings, particularly on discrete emotions, remains understudied. The study utilized 360-degree panoramic photo scene modeling to modify ceiling heights within virtual art galleries, assessing emotional responses through self-report measures. Participants were presented with virtual art gallery environments featuring varying ceiling heights. Two studies were conducted: Study 1 involved absolute emotion rating across different ceiling heights, and Study 2 focused on selecting ceiling heights based on assigned emotions. The data revealed that ceiling height significantly impacts specific emotions, notably disgust and joy. Lower ceiling heights generally evoked higher levels of fear and anger, while higher ceiling heights were associated with increased joy. The impact on other emotions like sadness, surprise, and disgust was more nuanced and varied across different ceiling heights. The findings highlight a complex relationship between ceiling height and emotional responses in art galleries. The study demonstrates the efficacy of using editable 360-degree VR panoramic scenes in environmental psychology and architecture research, offering insights into how spatial dimensions influence emotional experiences in architectural settings.

Panoramic Motion Perception Inspired Fly Visual Brain Joint Neural Network on Omnidirectional Collision Detection

Biological systems have a great number of visual motion detection neurons, some of which can preferentially react to specific visual regions. Nevertheless, little work has been performed about how they can be used to develop neural network models for omnidirectional collision detection. Hereby, an artificial fly visual brain neural network with presynaptic and postsynaptic subnetworks, for the first time, is developed to detect the changes of visual motion in panoramic scenes. Herein, the presynaptic subnetwork, which originates from the preferential response characteristics of five fly visual neurons, responds to all the moving objects in the panoramic field; the postsynaptic network, which is based on the properties of the angle and height detection neurons in the fly’s brain system, collects the excitatory intensities of the visual neurons, and outputs the real-time activities of the main object closest to the panoramic camera. Hereafter, a computational model is constructed to implement omnidirectional collision detection, relying upon the artificial visual brain neural network and three functional neurons. The theoretical analysis has verified that the collision detection model’s computational complexity depends mainly on the image input resolution. Three experimental conclusions can be clearly drawn: (i) the motion characteristics of the main object in the panoramic environment can be clearly exhibited in the neural network; (ii) the collision detection model can not only outperform the compared models but also successfully perform omnidirectional collision detection; and (iii) it spends 0.24 s or so to execute visual information processing per frame with the resolution of 120 × 80.

The Expressions of Political Power in Panoramic and Grand Space in Ancient Roman Empire before A.D. 476

This article argues the occupation, penetration and radiation of political power in the panoramic space by explaining the typical public architecture and public space in the former Roman Empire. As the most important public space, squares are the places where emperors show their powers. So the political elites try to occupy it. In Colosseum the emperors’ supreme power is thoroughly and incisively expressed by the carefully designed space settings. The public baths with compound decorations and functions are the visual panoramic scene. In this entertaining public space political power penetrates the individual life. The grandness of public architectures in roman city are the result of powers’ ostentation. With the imperial structure the styles and functions of public architectures in roman city radiates the whole empire. With these three relations between power and space this article draws two conclusions. First the ostentatious power based on the competitive power system is the connector between the republic and early empire. The great achievements in public architecture are the fruits of it. Second the super grand territory is the result of the links between military and politics. This links lead to the competitive power system. It also explains why the roman empire is invaded by the barbarians in the west but it continues in the east. 

Eye and head movements while encoding and recognizing panoramic scenes in virtual reality.

One approach to studying the recognition of scenes and objects relies on the comparison of eye movement patterns during encoding and recognition. Past studies typically analyzed the perception of flat stimuli of limited extent presented on a computer monitor that did not require head movements. In contrast, participants in the present study saw omnidirectional panoramic scenes through an immersive 3D virtual reality viewer, and they could move their head freely to inspect different parts of the visual scenes. This allowed us to examine how unconstrained observers use their head and eyes to encode and recognize visual scenes. By studying head and eye movement within a fully immersive environment, and applying cross-recurrence analysis, we found that eye movements are strongly influenced by the content of the visual environment, as are head movements-though to a much lesser degree. Moreover, we found that the head and eyes are linked, with the head supporting, and by and large mirroring the movements of the eyes, consistent with the notion that the head operates to support the acquisition of visual information by the eyes.

Eye Tracking in Virtual Reality.

This chapter explores the current state of the art in eye tracking within 3D virtual environments. It begins with the motivation for eye tracking in Virtual Reality (VR) in psychological research, followed by descriptions of the hardware and software used for presenting virtual environments as well as for tracking eye and head movements in VR. This is followed by a detailed description of an example project on eye and head tracking while observers look at 360° panoramic scenes. The example is illustrated with descriptions of the user interface and program excerpts to show the measurement of eye and head movements in VR. The chapter continues with fundamentals of data analysis, in particular methods for the determination of fixations and saccades when viewing spherical displays. We then extend these methodological considerations to determining the spatial and temporal coordination of the eyes and head in VR perception. The chapter concludes with a discussion of outstanding problems and future directions for conducting eye- and head-tracking research in VR. We hope that this chapter will serve as a primer for those intending to implement VR eye tracking in their own research.

Caste System Turns into A Social Curse and Social Discrimination: A Study of Mulk Raj Anand’s Untouchable in the Perspective of Post-independence Bangladesh

The caste system roots in the heart of Hinduism and falls apart Hindus into touchable and untouchable. The sweepers are regarded as untouchables and are given no choice and access to their social life. The caste system in Hinduism and, therefore, in the Hindu-majored nation in India is a strong social discriminatory hierarchy that has been exercised for more than two millenniums. Mulk Raj Anand, with a firm belief in the dignity and equality of all human beings, attempts to project a panoramic scene of the caste system by beckoning a single day from the diary of Bakha, an untouchable boy who is a sweeper in profession. The present paper attempts to address the curse and discrimination triggered by the caste system, which is prevalent in Mulk Raj Anand’s Untouchable. Though the scenes of the novel belong to a small, interior town in Punjab, the happenings are pan-Indian in nature. This paper also argues how the caste system paves the way for inter-caste conflict and exploitation and, apart from several caste discrimination, why changing the upper caste’ outlook is the sole way out to wipe out the stigma of the caste system.

View-Aware Salient Object Detection for $360^{\circ }$ Omnidirectional Image

Image-based salient object detection (ISOD) in <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> scenarios is significant for understanding and applying panoramic information. However, research on <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> ISOD has not been widely explored due to the lack of large, complex, high-resolution, and well-labeled datasets. Towards this end, we construct a large scale <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> ISOD dataset with object-level pixel-wise annotation on equirectangular projection (ERP), which contains rich panoramic scenes with not less than 2K resolution and is the largest dataset for <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> ISOD by far to our best knowledge. By observing the data, we find current methods face three significant challenges in panoramic scenarios: diverse distortion degrees, discontinuous edge effects and changeable object scales. Inspired by humans' observing process, we propose a view-aware salient object detection method based on a Sample Adaptive View Transformer (SAVT) module with two sub-modules to mitigate these issues. Specifically, the sub-module View Transformer (VT) contains three transform branches based on different kinds of transformations to learn various features under different views and heighten the model's feature toleration of distortion, edge effects and object scales. Moreover, the sub-module Sample Adaptive Fusion (SAF) is to adjust the weights of different transform branches based on various sample features and make transformed enhanced features fuse more appropriately. The benchmark results of 20 state-of-the-art ISOD methods reveal the constructed dataset is very challenging. Moreover, exhaustive experiments verify the proposed approach is practical and outperforms the state-of-the-art methods.

Weighting of Celestial and Terrestrial Cues in the Monarch Butterfly Central Complex.

Monarch butterflies rely on external cues for orientation during their annual long-distance migration from Northern US and Canada to Central Mexico. These external cues can be celestial cues, such as the sun or polarized light, which are processed in a brain region termed the central complex (CX). Previous research typically focused on how individual simulated celestial cues are encoded in the butterfly's CX. However, in nature, the butterflies perceive several celestial cues at the same time and need to integrate them to effectively use the compound of all cues for orientation. In addition, a recent behavioral study revealed that monarch butterflies can rely on terrestrial cues, such as the panoramic skyline, for orientation and use them in combination with the sun to maintain a directed flight course. How the CX encodes a combination of celestial and terrestrial cues and how they are weighted in the butterfly's CX is still unknown. Here, we examined how input neurons of the CX, termed TL neurons, combine celestial and terrestrial information. While recording intracellularly from the neurons, we presented a sun stimulus and polarized light to the butterflies as well as a simulated sun and a panoramic scene simultaneously. Our results show that celestial cues are integrated linearly in these cells, while the combination of the sun and a panoramic skyline did not always follow a linear integration of action potential rates. Interestingly, while the sun and polarized light were invariantly weighted between individual neurons, the sun stimulus and panoramic skyline were dynamically weighted when both stimuli were simultaneously presented. Taken together, this dynamic weighting between celestial and terrestrial cues may allow the butterflies to flexibly set their cue preference during navigation.

Scene Construction and Application of Panoramic Virtual Simulation in Interactive Dance Teaching Based on Artificial Intelligence Technology

In recent years, with the rapid development of human motion simulation technology and virtual simulation technology, natural human-computer interaction has become the main form of research in the computer industry. Due to the research capabilities and practical value of mobile video technology, it is widely used in advanced fields, such as video animation production, rehabilitation medicine, sports training, and game software, effectively realizing the connection between the three-dimensional world and the simulation world. However, the application of motion imaging technology in teaching activities is not perfect. This article is based on dance teaching, adopts a dance movement analysis method based on vector matching, and uses artificial intelligence technology to create a virtual panoramic scene. Experimental data show that the average recognition rate of gestures is 89.8%, which can meet the gesture switching interaction of working conditions in virtual simulation of equipment operation.

Quad‐fisheye Image Stitching for Monoscopic Panorama Reconstruction

AbstractMonoscopic panorama provides the display of omnidirectional contents surrounding the viewer. An increasingly popular way to reconstruct a panorama is to stitch a collection of fisheye images. However, such non‐planar views may result in problems such as distortions and boundary irregularities. In most cases, the computational expense for stitching non‐planar images is also too high to satisfy real‐time applications. In this paper, a novel monoscopic panorama reconstruction pipeline that produces better quad‐fisheye image stitching results for omnidirectional environment viewing is proposed. The main idea is to apply mesh deformation for image alignment. To optimize inter‐lens parallaxes, unwarped images are firstly cropped and reshuffled to facilitate the circular environment scene composition by the seamless ring‐connection of the panorama borders. Several mesh constraints are then adopted to ensure a high alignment accuracy. After alignment, the boundary of the result is rectified to be rectangular to prevent gapping artefacts. We further extend our approach to video stitching. The temporal smoothness model is added to prevent unexpected artefacts in the panoramic videos. To support interactive applications, our stitching algorithm is programmed using CUDA. The camera motion and average gradient per video frame are further calculated to accelerate for synchronous real‐life panoramic scene reconstruction and visualization. Experimental results demonstrate that our method has advantages in respects of alignment accuracy, adaptability and image quality of the stitching result.

High-performance panoramic annular lens design for real-time semantic segmentation on aerial imagery

As the visual perception window of the drone system, the lens provides great help for obtaining visual information, detection, and recognition. However, traditional lenses carried on drones cannot have characteristics of a large field of view (FoV), small size, and low weight at the same time. To meet the above requirements, we propose a panoramic annular lens (PAL) system with 4K high resolution, a large FoV of (30 deg to 100 deg) × 360 deg, an angular resolution of 12.2 mrad of aerial perspective, and great imaging performance. We equip a drone system with our designed PAL to collect panoramic image data at an altitude of 100 m from the track and field and obtain the first drone-perspective panoramic scene segmentation dataset Aerial-PASS, with annotated labels of track and field. We design an efficient deep architecture for aerial scene segmentation. Trained on Aerial-PASS, the yielded model accurately segments aerial images. Compared with the ERF-PAPNet and SwiftNet semantic segmentation networks, the network we adopted has higher recognition accuracy with the mean IoU greater than 86.30%, which provides an important reference for the drone system to monitor and identify specific targets in real-time in a large FoV.

See360: Novel Panoramic View Interpolation.

We present See360, which is a versatile and efficient framework for 360° panoramic view interpolation using latent space viewpoint estimation. Most of the existing view rendering approaches only focus on indoor or synthetic 3D environments and render new views of small objects. In contrast, we suggest to tackle camera-centered view synthesis as a 2D affine transformation without using point clouds or depth maps, which enables an effective 360° panoramic scene exploration. Given a pair of reference images, the See360 model learns to render novel views by a proposed novel Multi-Scale Affine Transformer (MSAT), enabling the coarse-to-fine feature rendering. We also propose a Conditional Latent space AutoEncoder (C-LAE) to achieve view interpolation at any arbitrary angle. To show the versatility of our method, we introduce four training datasets, namely UrbanCity360, Archinterior360, HungHom360 and Lab360, which are collected from indoor and outdoor environments for both real and synthetic rendering. Experimental results show that the proposed method is generic enough to achieve real-time rendering of arbitrary views for all four datasets. In addition, our See360 model can be applied to view synthesis in the wild: with only a short extra training time (approximately 10 mins), and is able to render unknown real-world scenes. The superior performance of See360 opens up a promising direction for camera-centered view rendering and 360° panoramic view interpolation.

Mosaic: Advancing User Quality of Experience in 360-Degree Video Streaming With Machine Learning

Conventional streaming solutions for streaming 360-degree panoramic videos are inefficient in that they download the entire 360-degree panoramic scene, while the user views only a small sub-part of the scene called the viewport. This can waste over 80% of the network bandwidth. We develop a comprehensive approach called Mosaic that combines a powerful neural network-based viewport prediction with a rate control mechanism that assigns rates to different tiles in the 360-degree frame such that the video quality of experience is optimized subject to a given network capacity. We model the optimization as a multi-choice knapsack problem and solve it using a greedy approach. We also develop an end-to-end testbed using standards-compliant components and provide a comprehensive performance evaluation of Mosaic along with five other streaming techniques – two for conventional adaptive video streaming and three for 360-degree tile-based video streaming. Mosaic outperforms the best of the competitions by as much as 47-191% in terms of average video quality of experience. Simulation-based evaluation as well as subjective user studies further confirm the superiority of the proposed approach.

Influence of field of view in visual prostheses design: Analysis with a VR system

In this work, we evaluate the influence of field of view with respect to spatial resolution in visual prostheses. Our system uses a virtual-reality environment based on panoramic scenes and headmounted display which allows users to feel immersed in the scene. Results indicate that, for the design of retinal prostheses, it is better to concentrate the phosphenes in a small area, to maximize the angular resolution, even if that implies sacrificing field of view.

The Antarium: A Reconstructed Visual Reality Device for Ant Navigation Research.

We constructed a large projection device (the Antarium) with 20,000 UV-Blue-Green LEDs that allows us to present tethered ants with views of their natural foraging environment. The ants walk on an air-cushioned trackball, their movements are registered and can be fed back to the visual panorama. Views are generated in a 3D model of the ants’ environment so that they experience the changing visual world in the same way as they do when foraging naturally. The Antarium is a biscribed pentakis dodecahedron with 55 facets of identical isosceles triangles. The length of the base of the triangles is 368 mm resulting in a device that is roughly 1 m in diameter. Each triangle contains 361 blue/green LEDs and nine UV LEDs. The 55 triangles of the Antarium have 19,855 Green and Blue pixels and 495 UV pixels, covering 360° azimuth and elevation from −50° below the horizon to +90° above the horizon. The angular resolution is 1.5° for Green and Blue LEDs and 6.7° for UV LEDs, offering 65,536 intensity levels at a flicker frequency of more than 9,000 Hz and a framerate of 190 fps. Also, the direction and degree of polarisation of the UV LEDs can be adjusted through polarisers mounted on the axles of rotary actuators. We build 3D models of the natural foraging environment of ants using purely camera-based methods. We reconstruct panoramic scenes at any point within these models, by projecting panoramic images onto six virtual cameras which capture a cube-map of images to be projected by the LEDs of the Antarium. The Antarium is a unique instrument to investigate visual navigation in ants. In an open loop, it allows us to provide ants with familiar and unfamiliar views, with completely featureless visual scenes, or with scenes that are altered in spatial or spectral composition. In closed-loop, we can study the behavior of ants that are virtually displaced within their natural foraging environment. In the future, the Antarium can also be used to investigate the dynamics of navigational guidance and the neurophysiological basis of ant navigation in natural visual environments.

The role of reference frame in panoramic scene memory

The role of reference frame in panoramic scene memory