Panoramic Video Research Articles

Pipeline 3D Modeling Based on High-Definition Rendering Intelligent Calculation

In the processing of panoramic video, projection mapping is a very critical step. The selection of the projection mapping format will affect the performance, transmission mode, and rendering mode of the panoramic video codec. Therefore, this article starts from the projection mapping format, analyzes the mapping process of the standard mapping format, and then proposes a method of rendering panoramic video in the projection mapping format. By analyzing the parallel design schemes of swarm intelligence algorithms under different granularities, this paper proposes a parallel swarm intelligence optimization algorithm design method and then designs and implements a parallel artificial bee colony algorithm. With the help of the ArcGIS Engine development platform, this paper defines the interface for data exchange. With the support of Multipatch format data in ArcGIS software, through secondary development, the three-dimensional pipeline automatic modeling module is established, and the pipeline model is automatically generated. The digital construction and visualization of the company play a driving role. Based on the understanding of the characteristics of the pipeline image itself, combined with the analysis of the shortcomings of the existing methods, this paper proposes a new deep learning-based high-definition rendering solution for the pipeline image. In this paper, the pipeline image is preprocessed, and then the processed pipeline image is converted into a style pipeline image through the pipeline image style transfer technology, and the obtained style pipeline image is postprocessed to enhance the effect. The preprocessing of pipeline images mainly includes pipeline image enhancement and pipeline image filtering operations. Its purpose is to change the distribution of pipeline images to improve the quality of pipeline images and make them more suitable for subsequent style conversion. In the part of pipeline image style conversion, this paper proposes a new deep learning-based pipeline image high-definition rendering network, which consists of three subnetworks: pipeline image feature modeling module, feature model alignment module, and pipeline image re-rendering module. This article has conducted sufficient experiments to fully compare the processing results of the method proposed in this article and other existing methods and at the same time shows the high-quality high-definition rendering results. The experimental results verify the excellent performance of the method proposed in this paper.

Simulated visual hallucinations in virtual reality enhance cognitive flexibility

Historically, psychedelic drugs are known to modulate cognitive flexibility, a central aspect of cognition permitting adaptation to changing environmental demands. Despite proof suggesting phenomenological similarities between artificially-induced and actual psychedelic altered perception, experimental evidence is still lacking about whether the former is also able to modulate cognitive flexibility. To address this, we measure participants’ cognitive flexibility through behavioral tasks after the exposure to virtual reality panoramic videos and their hallucinatory-like counterparts generated by the DeepDream algorithm. Results show that the estimated semantic network has a flexible structure when preceded by altered videos. Crucially, following the simulated psychedelic exposure, individuals also show an attenuated contribution of the automatic process and chaotic dynamics underlying the decision process. This suggests that simulated altered perceptual phenomenology enhances cognitive flexibility, presumably due to a reorganization in the cognitive dynamics that facilitates the exploration of uncommon decision strategies and inhibits automated choices.

The effects of virtual reality tourism involvement on place attachment and behavioral intentions: virtual reality tourism of the Yellow Crane Tower in Wuhan

ABSTRACT The main purpose of this research was to explore the relationships among virtual reality tourism involvement (VRTI), place attachment, and behavioral intentions. Based on involvement theory. The VR panoramic video of the Yellow Crane Tower in Wuhan, Hubei Province, China was the context, and 693 valid questionnaires were analyzed through AMOS 24.0. VRTI (pleasure, sign, and centrality) had significant positive effects on place attachment. Pleasure and sign had significant positive influences on behavioral intentions, while centrality had a significant negative effect. Place identity had a significant positive effect on behavioral intentions, whereas place dependence had no effect. Place dependence had a significant positive effect on place identity. Place dependence had a partial mediating effect between pleasure, centrality, and behavioral intentions, as well as a complete mediating effect between sign and behavioral intentions. Place identity played a partial mediating role between pleasure, centrality, and sign and behavioral intentions. VRTI explores a novel human-place relationship of “virtual tourism destinations – tourists – real tourism destinations”. This study expands the research on place attachment and tourist behavioral decisions, enriches the contents by applying involvement theory, proposes management strategies for VR tourism development, and provides new insights for destination marketing.

Sports Video Panorama Synthesis Technology Based on Edge Computing and Video Shot Boundary Detection

The development requirements for sports panorama synthesis technology, which rely on modern network technology, abandon traditional basketball training forms, and make effective use of the application and development of video panorama technology in the actual training process, are not only an important response to the current characteristics of students’ physical education learning and physical and mental growth but also an important response to the current characteristics of students’ physical education learning and physical and mental growth. In sports video analysis technology, a sports video panorama is a technical tool that converts an action video into a static action panorama to achieve the effect of action freezing and make overall analysis and mastery of the action easier. Shot segmentation is the foundation of hierarchical video structure, and it necessitates the accurate detection of all types of complex edited shot boundaries, as well as the effective distinction of motion changes in shots, in order to avoid shot boundary recognition being hampered. The synthesis technology for sports video panorama is investigated in this paper using edge computing and video shot boundary detection. After obtaining the boundary feature that describes the video shot, a comparison of this feature with a predetermined threshold value can be used to determine whether there is shot shear.

Viewport-Driven Adaptive 360◦ Live Streaming Optimization Framework

Virtual reality (VR) video streaming and 360◦ panoramic video have received extensive attention in recent years, which can bring users an immersive experience. However, the ultra-high bandwidth and ultra-low latency requirements of virtual reality video or 360◦ panoramic video also put tremendous pressure on the carrying capacity of the current network. In fact, since the user’s field of view (a.k.a viewport) is limited when watching a panoramic video and users can only watch about 20%∼30% of the video content, it is not necessary to directly transmit all high-resolution content to the user. Therefore, predicting the user’s future viewing viewport can be crucial for selective streaming and further bitrate decisions. Combined with the tile-based adaptive bitrate (ABR) algorithm for panoramic video, video content within the user’s viewport can be transmitted at a higher resolution, while areas outside the viewport can be transmitted at a lower resolution. This paper mainly proposes a viewport-driven adaptive 360◦ live streaming optimization framework, which combines viewport prediction and ABR algorithm to optimize the transmission of live 360◦ panoramic video. However, existing viewport prediction always suffers from low prediction accuracy and does not support real-time performance. With the advantage of convolutional network (CNN) in image processing and long short-term memory (LSTM) in temporal series processing, we propose an online-updated viewport prediction model called LiveCL which mainly utilizes CNN to extract the spatial characteristics of video frames and LSTM to learn the temporal characteristics of the user’s viewport trajectories. With the help of the viewport prediction and ABR algorithm, unnecessary bandwidth consumption can be effectively reduced. The main contributions of this work include: (1) a framework for 360◦ video transmission is proposed; (2) an online real-time viewport prediction model called LiveCL is proposed to optimize 360◦ video transmission combined with a novel ABR algorithm, which outperforms the existing model. Based on the public 360◦ video dataset, the tile accuracy, recall, precision, and frame accuracy of LiveCL are better than those of the latest model. Combined with related adaptive bitrate algorithms, the proposed viewport prediction model can reduce the transmission bandwidth by about 50%.

Prediction of Motion Sickness Degree of Stereoscopic Panoramic Videos Based on Content Perception and Binocular Characteristics

Prediction of Motion Sickness Degree of Stereoscopic Panoramic Videos Based on Content Perception and Binocular Characteristics

Egocentric Two-frame Pedestrian Trajectory Prediction Algorithm Based on a Panoramic Camera

With the development of service industry, service-oriented mobile robot navigation technology has received more and more attention. In order to improve the obstacle avoidance performance of service-oriented mobile robots, accurate pedestrian trajectory prediction is essential. Therefore, we propose an egocentric two-frame pedestrian trajectory prediction algorithm based on a panoramic camera, the Future Social Prediction (FSP) algorithm, to solve the pedestrian trajectory prediction problem in the human-robot coexistence environment. The input of FSP is the information in the two frames of a panoramic camera. First, the free pedestrian prediction network (FPPN) is used to predict the free movement of pedestrians. Then the future social pooling module (FSPM) is used to process the prediction results of FPPN. And finally, the social pedestrian prediction network (SPPN) combines the frame information with the output of FSPM to get the final prediction results. A panoramic camera is used as a sensor to reduce blind spots in the field of view. Only two frames are used for prediction to improve the real-time performance of FSP. We record the panoramic video and create a dataset. FSP is tested on our dataset with other two algorithms, and the results show that FSP has better performance than Spatio-Temporal Encoder-Decoder model (STED) and Future Person Localization (FPL).

Deep Learning Technique for Object Detection from Panoramic Video Frames

Deep Learning Technique for Object Detection from Panoramic Video Frames

Feasibility Study on Using AI and VR for Decision-Making Training of Basketball Players

Decision-making plays an essential role in basketball offense. Offensive players must make effective decisions to score a basket in various defensive situations. Virtual reality (VR) has been widely used in the training of athletes to strengthen their ability to make optimal decisions by creating controllable repeatable training scenarios. In this article, an action-aware offensive decision-making training system for basketball using VR and artificial intelligence is proposed. The proposed system is composed of different virtual defensive scenarios and an offensive action recognition framework. Trainees wearing head-mounted display and a motion capture suit are trained by intuitively interacting with the VR system and receive decision suggestions when a bad one is made. This study changes the training media and methods to create an immersive training environment during the training phase and evaluates the training effectiveness. These training scenarios are a conventional tactics board, the proposed VR system with a prerecorded <inline-formula><tex-math notation="LaTeX">$360^\circ$</tex-math></inline-formula> panorama video, and the VR system with computer-simulated virtual content. Results indicate that the training scenario affects the training in terms of decision time.

Correlations Between Panoramic Imagery and Gamma-Ray Background in an Urban Area

When searching for radiological sources in an urban area, a vehicle-borne detector system will often measure complex, varying backgrounds primarily from natural gamma-ray sources. Much work has been focused on developing spectral algorithms that retain sensitivity and minimize the false-positive rate even in the presence of such spectral and temporal variability. However, information about the environment surrounding the detector system might also provide useful clues about the expected background, which if incorporated into an algorithm, could improve performance. Recent work has focused on extensive measuring and modeling of urban areas with the goal of understanding how these complex backgrounds arise. This work presents an analysis of panoramic video images and gamma-ray background data collected in Oakland, California, by the radiological multisensor analysis platform (RadMAP) vehicle. Features were extracted from the panoramic images by semantically labeling the images and then convolving the labeled regions with the detector response. A linear model was used to relate the image-derived features to gamma-ray spectral features obtained using nonnegative matrix factorization (NMF) under different regularizations. We find some gamma-ray background features correlate strongly with image-derived features that measure the response-adjusted solid angle subtended by sky and buildings, and we discuss the implications for the development of future, contextually aware detection algorithms.

Research on the Application of Environmental Art Design Based on the Combination of VR and Panoramic Video Technology

The panoramic video technology is introduced to collect multiangle data of design objects, draw a 3D spatial model with the collected data, solve the first-order differential equation for the 3D spatial model, obtain the spatial positioning extremes of the object scales, and realize the alignment and fusion of panoramic video images according to the positioning extremes above and below the scale space. Then, the panoramic video is generated and displayed by computer processing so that the tourist can watch the scene with virtual information added to the panoramic video by wearing the display device elsewhere. It solves the technical difficulties of the high complexity of the algorithm in the system of panoramic video stitching and the existence of stitching cracks and the “GHOST” phenomenon in the stitched video, as well as the technical difficulties that the 3D registration is easily affected by the time-consuming environment and target tracking detection algorithm. The simulation results show that the panoramic video stitching method performs well in real time and effectively suppresses stitching cracks and the “GHOST” phenomenon, and the augmented reality 3D registration method performs well for the local enhancement of the panoramic video.

Virtual look around: comparing presence, cybersickness and usability for virtual tours across different devices

Virtual Reality has become readily available in the last few years through different devices, from desktop computers to head­-mounted displays (HMD). Also, virtual tours became popular with 360º panoramic photographs and video clips on online social media, so people could visit remote locations without being exposed to crowded transportation or long travels. Also, virtual tours demonstrate considerable potential as a form of escapism and even for remote teaching. Since we lack studies that evaluate the User Experience (UX) in virtual tours on different devices, this article aims to compare aspects of the User Experience (regarding sense of presence, cybersickness, and usability) in a virtual tour website developed in WebXR across different devices. To achieve our objective, we developed a virtual tour based on 360º pictures using WebXR API and React 360 framework and conducted an experiment with 41 undergraduate students using four different devices: a laptop computer, a smartphone, a Google Cardboard headset, and a Samsung Gear VR HMD. We evaluated users’ perceptions by adapting and translating the Suitability Evaluation Questionnaire (SEQ) and users’ performance by measuring the time to fulfill a set of tasks. The main findings from this study include that (i) the overall self-­reported experience using Google Cardboard is worse than using other devices, (ii) the users’ performance is quite similar between the platforms, (iii) there is evidence of unexpected cybersickness symptoms in tests with the smartphone, and (iv) the development of a plausible hypothesis concerning low usability having an effect upon the sense of presence. Additional contributions of our research are the adaptation, translation into Portuguese, psychometric analysis, and revised scoring procedures of the SEQ.

Pedestrian-Aware Panoramic Video Stitching Based on a Structured Camera Array

The panorama stitching system is an indispensable module in surveillance or space exploration. Such a system enables the viewer to understand the surroundings instantly by aligning the surrounding images on a plane and fusing them naturally. The bottleneck of existing systems mainly lies in alignment and naturalness of the transition of adjacent images. When facing dynamic foregrounds, they may produce outputs with misaligned semantic objects, which is evident and sensitive to human perception. We solve three key issues in the existing workflow that can affect its efficiency and the quality of the obtained panoramic video and present Pedestrian360, a panoramic video system based on a structured camera array (a spatial surround-view camera system). First, to get a geometrically aligned 360○ view in the horizontal direction, we build a unified multi-camera coordinate system via a novel refinement approach that jointly optimizes camera poses. Second, to eliminate the brightness and color difference of images taken by different cameras, we design a photometric alignment approach by introducing a bias to the baseline linear adjustment model and solving it with two-step least-squares. Third, considering that the human visual system is more sensitive to high-level semantic objects, such as pedestrians and vehicles, we integrate the results of instance segmentation into the framework of dynamic programming in the seam-cutting step. To our knowledge, we are the first to introduce instance segmentation to the seam-cutting problem, which can ensure the integrity of the salient objects in a panorama. Specifically, in our surveillance oriented system, we choose the most significant target, pedestrians, as the seam avoidance target, and this accounts for the name Pedestrian360 . To validate the effectiveness and efficiency of Pedestrian360, a large-scale dataset composed of videos with pedestrians in five scenes is established. The test results on this dataset demonstrate the superiority of Pedestrian360 compared to its competitors. Experimental results show that Pedestrian360 can stitch videos at a speed of 12 to 26 fps, which depends on the number of objects in the shooting scene and their frequencies of movements. To make our reported results reproducible, the relevant code and collected data are publicly available at https://cslinzhang.github.io/Pedestrian360-Homepage/ .

Surface kinematic and depth-resolved analysis of human vocal folds in vivo during phonation using optical coherence tomography.

.Significance: The human vocal fold (VF) oscillates in multiple vectors and consists of distinct layers with varying viscoelastic properties that contribute to the mucosal wave. Office-based and operative laryngeal endoscopy are limited to diagnostic evaluation of the VF epithelial surface only and are restricted to axial-plane characterization of the horizontal mucosal wave. As such, understanding of the biomechanics of human VF motion remains limited.Aim: Optical coherence tomography (OCT) is a micrometer-resolution, high-speed endoscopic imaging modality which acquires cross-sectional images of tissue. Our study aimed to leverage OCT technology and develop quantitative methods for analyzing the anatomy and kinematics of in vivo VF motion in the coronal plane.Approach: A custom handheld laryngeal stage was used to capture OCT images with 800 A-lines at 250 Hz. Automated image postprocessing and analytical methods were developed.Results: Novel kinematic analysis of in vivo, long-range OCT imaging of the vibrating VF in awake human subjects is reported. Cross-sectional, coronal-plane panoramic videos of the larynx during phonation are presented with three-dimensional videokymographic and space-time velocity analysis of VF motion.Conclusions: Long-range OCT with automated computational methods allows for cross-sectional dynamic laryngeal imaging and has the potential to broaden our understanding of human VF biomechanics and sound production.

Assessing Emotional Responses to the Spatial Quality of Urban Green Spaces through Self-Report and Face Recognition Measures.

Although creating a high-quality urban green space (UGS) is of considerable importance in public health, few studies have used individuals’ emotions to evaluate the UGS quality. This study aims to conduct a multidimensional emotional assessment method of UGS from the perspective of spatial quality. Panoramic videos of 15 scenes in the West Lake Scenic Area were displayed to 34 participants. For each scene, 12 attributes regarding spatial quality were quantified, including perceived plant attributes, spatial structure attributes, and experiences of UGS. Then, the Self-Assessment-Manikin (SAM) scale and face recognition model were used to measure people’s valence-arousal emotion values. Among all the predictors, the percentages of water and plants were the most predictive indicators of emotional responses measured by SAM scale, while the interpretation rate of the model measured by face recognition was insufficiently high. Concerning gender differences, women experienced a significantly higher valence than men. Higher percentages of water and plants, larger sizes, approximate shape index, and lower canopy densities were often related to positive emotions. Hence, designers must consider all structural attributes of green spaces, as well as enrich visual perception and provide various activities while creating a UGS. In addition, we suggest combining both physiological and psychological methods to assess emotional responses in future studies. Because the face recognition model can provide objective measurement of emotional responses, and the self-report questionnaire is much easier to administer and can be used as a supplement.

Strategies for generating panoramic video images without information about scene correspondences for multispectral distributed aperture systems

We derive analytical expressions for calculating the number of elementary computational operations required to generate several personal regions of interest in a panoramic computer-vision distributed-aperture system using two alternative strategies: strategy 1 involves acquisition of a complete panoramic frame, followed by the selection of personal regions of interest, while with strategy 2 the region of interest is directly formed for each user. The parameters of analytical expressions include the number of cameras in the distributed system, the number of users, and the resolution of panorama and user frames. The formulas obtained for the given parameters make it possible to determine a strategy that would be optimal in terms of a criterion of the minimum number of elementary computational operations for generating multiple personal regions of interest. The region of interest is generated using only a priori information about the internal and external camera parameters, obtained as a result of their photogrammetric calibration with a universal test object, and does not take into account information about scene correspondences at the boundaries of intersecting fields of view.

10.4: A panoramic video processing algorithm based on variable surface projection

10.4: A panoramic video processing algorithm based on variable surface projection

A Standardized Workflow Based on the STAVIRO Unbaited Underwater Video System for Monitoring Fish and Habitat Essential Biodiversity Variables in Coastal Areas

Essential Biodiversity Variables (EBV) related to benthic habitats and high trophic levels such as fish communities must be measured at fine scale but monitored and assessed at spatial scales that are relevant for policy and management actions. Local scales are important for assessing anthropogenic impacts, and conservation-related and fisheries management actions, while reporting on the conservation status of biodiversity to formulate national and international policies requires much broader scales. Measurements must account for the fact that coastal habitats and fish communities are heterogeneously distributed locally and at larger scales. Assessments based on in situ monitoring generally suffer from poor spatial replication and limited geographical coverage, which is challenging for area-wide assessments. Requirements for appropriate monitoring comprise cost-efficient and standardized observation protocols and data formats, spatially scalable and versatile data workflows, data that comply with the FAIR (Findable, Accessible, Interoperable, and Reusable) principles, while minimizing the environmental impact of measurements. This paper describes a standardized workflow based on remote underwater video that aims to assess fishes (at species and community levels) and habitat-related EBVs in coastal areas. This panoramic unbaited video technique was developed in 2007 to survey both fishes and benthic habitats in a cost-efficient manner, and with minimal effect on biodiversity. It can be deployed in areas where low underwater visibility is not a permanent or major limitation. The technique was consolidated and standardized and has been successfully used in varied settings over the last 12 years. We operationalized the EBV workflow by documenting the field protocol, survey design, image post-processing, EBV production and data curation. Applications of the workflow are illustrated here based on some 4,500 observations (fishes and benthic habitats) in the Pacific, Indian and Atlantic Oceans, and Mediterranean Sea. The STAVIRO’s proven track-record of utility and cost-effectiveness indicates that it should be considered by other researchers for future applications.

EPASS360: QoE-Aware 360-Degree Video Streaming Over Mobile Devices

The 360-degree video streaming system delivers a monocular panoramic video surrounding the user, and the user can change the viewing direction of mobile devices to see different parts of the video through the “viewport”. Due to the limited network bandwidth, playbacks of high-resolution 360-degree videos often suffer from rebuffering, while too much bandwidth is wasted in delivering those out-of-viewport parts that the user never watches. In this article, we present an Ensemble Prediction and Allocation based Streaming System, named as EPASS360, for delivering high Quality of Experience (QoE) 360-degree videos. The prediction model takes advantages of ensemble learning, providing high accuracy on the prediction of viewports. The allocation model divides a video into tiles, and allocates high resolution to tiles where a user's viewpoint may appear in the future by solving the QoE-aware optimization problem. Trace-driven emulation on real-world datasets shows that EPASS360 enhances the QoE in various scenarios compared to state-of-the-art streaming approaches. Experiments on the head-mounted device and the hand-held device over real-world Internet confirm the high user experience of EPASS360.

Real-Time Video Scan Method for Community Partner Use to Inform Play Streets Implementation.

System for Observing Play and Recreation in Communities (SOPARC) can provide accurate assessment of physical activity; however, the skills, time, and human resources necessary to collect/interpret SOPARC data can be challenging for community organizations. This article describes a more accessible adaptation of SOPARC using video recordings for community organizations to obtain physical activity feedback at Play Streets. Narrated panoramic video scans occurred every 30 minutes at each Play Street using an iPad. Videographers narrated: (1) sex, (2) age group (child, teen, adult, senior), and (3) activity level (sedentary, walking, vigorous) for everyone recorded. SOPARC video scans, in-person iSOPARC observations, and interviews were conducted with Play Streets implementors to determine validity and feasibility. Validity was examined using Lin's concordance correlation coefficient (CCC). In-person and video scans showed near perfect agreement for sedentary individuals (CCC = .95) and substantial agreement for active individuals (CCC = .72). Overall, community partners felt that they "could see how [the scans] could be useful" and "help[ed] see a bit more clearly what's happening." The method described here is a more accessible systematic observation approach to measure physical activity for communities implementing Play Streets. Further, this method can be used without research training while still providing valuable activity feedback.