Panoramic Video Research Articles

Understanding how virtual reality forest experience promote physiological and psychological health for patients undergoing hemodialysis.

Growing research has found that exposing patients to forest landscapes through technology improves their health. However, studies on the effects of nature therapy sessions on particularly vulnerable groups that require frequent regular treatment such as patients of chronic kidney disease (CKD) is lacking. This study aims to understand the effects of watching panoramic videos of forest walks through virtual reality (VR) headsets on patients. We also examine the effects of the frequency of virtual exposure to forests on the physiological and psychological parameters of patients undergoing dialysis. Twenty-four dialysis patients with a mean age of 65.11 year underwent a virtual nature intervention over the course of 3 weeks. The intervention consists of 3 numbers of 6-min 360-degree nature videos delivered using VR headsets. We measured heart rate variability (HRV) and heart rate (HR) using continuous electrocardiographic (ECG) monitoring and assessed their emotional states using a questionnaire survey and in-depth interviews. The results showed that the initial 6 min of watching of the panoramic videos through VR headsets resulted in significantly reduced HR and sympathetic nerve activity and increased positive emotional responses. However, repeated VR exposure did not further improve the physiological health of the respondents. Considering these results, the use of VR headsets to watch 6-min nature videos indoors can be used to enhance the positive psychological responses among dialysis and bed-ridden patients. As this study is preliminary, future random controlled trials are needed to compare and determine the best dose, content, and delivery methods of such an intervention.

Federated Deep Reinforcement Learning-Based Caching and Bitrate Adaptation for VR Panoramic Video in Clustered MEC Networks

Virtual reality (VR) panoramic video is more expressive and experiential than traditional video. With the accelerated deployment of 5G networks, VR panoramic video has experienced explosive development. The large data volume and multi-viewport characteristics of VR panoramic videos make it more difficult to cache and transcode them in advance. Therefore, VR panoramic video services urgently need to provide powerful caching and computing power over the edge network. To address this problem, this paper establishes a hierarchical clustered mobile edge computing (MEC) network and develops a data perception-driven clustered-edge transmission model to meet the edge computing and caching capabilities required for VR panoramic video services. The joint optimization problem of caching and bitrate adaptation can be formulated as a Markov Decision Process (MDP). The federated deep reinforcement learning (FDRL) algorithm is proposed to solve the problem of caching and bitrate adaptation (called FDRL-CBA) for VR panoramic video services. The simulation results show that FDRL-CBA can outperform existing DRL-based methods in the same scenarios in terms of cache hit rate and quality of experience (QoE). In conclusion, this work developed a FDRL-CBA algorithm based on a data perception-driven clustered-edge transmission model, called Hierarchical Clustered MEC Networks. The proposed method can improve the performance of VR panoramic video services.

Validation of a surgical workspace scale during robot-assisted surgery.

A sufficient surgical workspace is crucial to avoid complications. Within classic laparoscopy, many subjective surgical rating scales (SRSs) have previously been used to evaluate the surgical workspace. This study aimed to validate a modified version of the 5-point SRS during robot-assisted radical nephrectomy (RARN). Thirty-two intra-operative videos of intraperitoneal spaces were recorded from eight patients who underwent RARN. To attain the visualisation of different types of workspaces, we recorded 20 s panoramic videos of different pneumoperitoneum, namely 3, 5, 7 and 12 mmHg. The videos were randomised and presented two times to eight experienced robotic surgeons to evaluate the workspace using our modified 5-point SRS. Both inter-and intra-rater reliabilities were tested. The results of the validation study showed moderate inter-rater and good to excellent intra-rater reliability. This is a valid tool that can be confidently used by future researchers in the field of robot-assisted surgery.

Time-based Calibration: A Way to Ensure that Stitched Images are Captured Simultaneously

<p>With the rapid development of modern science and technology, people’s demand for such information as images and videos is also growing, and the requirements for image and video quality, clarity and view range are also increasing. Therefore, the construction of high-resolution, wide-view panoramic video has also gradually become a hot research topic. Video stitching technology is an increasingly popular research direction in the field of graphics, and the approach addresses the problem of a limited range of views due to a single device capture. Many researchers have proposed various algorithms for video stitching and achieved good stitching results, but the research on time synchronization calibration of different video sources is not yet well developed. This thesis proposes a multi-source video frames calibration technique based on external information sources for solving the problems of ghosting and cutting when stitching with different video sources. The proposed method calibrates the video stitching by introducing an information source and calculating the time difference of different devices. The error of the calibrated video stitching is less than 33 ms, which can guarantee the quality of the spliced video.</p> <p> </p>

Prediction of motion sickness degree of stereoscopic panoramic videos based on content perception and binocular characteristics

The immersive viewing environment of Virtual Reality (VR) system inevitably brings motion sickness to users, and restricts the development of VR system to a certain extent. In order to solve the problem of motion sickness when users view, it is necessary to predicate the motion sickness degree of the visual content. However, the methods based on subjective questionnaires or physiological signals will become cumbersome and infeasible with the growth of demand and supply of VR content. Most of the existing prediction methods for motion sickness are designed for 2D panoramic video. In this paper, a motion sickness prediction method based on content perception and binocular characteristics is proposed for stereoscopic panoramic video. The proposed method mainly consists of two modules: feature extraction based on content analysis and time pooling. The feature extraction module takes into account the attention mechanism of human visual system and the multi-channel characteristics of retina, and simulates the sensory conflict and the three-stage process of binocular stereoscopic perception in virtual environment. The time pooling module takes into account the time memory effect of human eyes. The proposed method has achieved excellent prediction performance on stereoscopic panoramic video comfort database SPVCD, its predication results have a good correlation with the subjective scores, which shows that the proposed method can effectively predict the motion sickness degree of VR content.

Emotional-Health-Oriented Urban Design: A Novel Collaborative Deep Learning Framework for Real-Time Landscape Assessment by Integrating Facial Expression Recognition and Pixel-Level Semantic Segmentation.

Emotional responses are significant for understanding public perceptions of urban green space (UGS) and can be used to inform proposals for optimal urban design strategies to enhance public emotional health in the times of COVID-19. However, most empirical studies fail to consider emotion-oriented landscape assessments under dynamic perspectives despite the fact that individually observed sceneries alter with angle. To close this gap, a real-time sentimental-based landscape assessment framework is developed, integrating facial expression recognition with semantic segmentation of changing landscapes. Furthermore, a case study using panoramic videos converted from Google Street View images to simulate changing scenes was used to test the viability of this framework, resulting in five million big data points. The result of this study shows that through the collaboration of deep learning algorithms, finer visual variables were classified, subtle emotional responses were tracked, and better regression results for valence and arousal were obtained. Among all the predictors, the proportion of grass was the most significant predictor for emotional perception. The proposed framework is adaptable and human-centric, and it enables the instantaneous emotional perception of the built environment by the general public as a feedback survey tool to aid urban planners in creating UGS that promote emotional well-being.

Stationary wavelet transformation based video stabilization and stitching

Video stitching has become popular due to recent advancements in technology to provide broad views and high-resolution displays. Comprehensive view or panoramic videos and high-resolution displays are created by stitching videos captured by multiple cameras or by a single camera at different points of time. This paper proposes a video stitching technique with stabilization for moving multi-camera videos adopting the wavelet decomposition technique. This method uses only those feature points that reduce the mismatching and increase the precision in estimating the transformation from among the feature points identified by the Speed-Up Robust Features detector. This work differs from the similar work of others in two directions. Instead of using all selected feature points for the matching purpose, only significant among them are used. Unlike others, the frames are stabilized before they are stitched.

An Overview of Panoramic Video Projection Schemes in the IEEE 1857.9 Standard for Immersive Visual Content Coding

Panoramic video contains 360-degree video content and it is convenient to draw the content that the user wants to watch on the head-Mount display (HMD) of the virtual reality display device. However, few encoders code panoramic video directly. Panoramic video content needs to be projected to a compression-friendly rectangular plane first. Then the existing video coding standards such as AVC, HEVC, VVC, AVS are used to code the panoramic video content. In order to improve the panoramic video coding efficiency, IEEE 1857.9 immersive video content coding working group was established to develop efficient immersive video projection and coding methods. This paper presents an overview of the panoramic video projection schemes in Immersive Visual Content Coding Standard IEEE 1857.9.

A panoramic photo mosaic method based on asymmetric two-way flow field

The main problems of panoramic photo stitching are discontinuity of stitching area and ghost, which has a great impact on the realization of panoramic video stitching. A panoramic photo mosaic method for asymmetric two-way flow field is proposed. In the analysis of asymmetric two-way flow field technology, through optical flow motion analysis and virtual close computer vision. Based on this technology, according to the similarity and smoothness between images, the discontinuous method is used to estimate the fusion area and realize image fusion processing. Considering that the image gray space contains a large amount of smoothing information, one-step method is selected as the flow field estimation method, and the panoramic photo mosaic is realized by combining the occlusion calculation and flow field estimation results. The empirical results show that the average stitching time of 10 panoramic images per frame is 21.588 s, and the minimum and maximum resolution of panoramic photos are 4040 respectively × 1080 dpi and 4580 × 1080 dpi. Subjective evaluation, objective evaluation and fuzzy evaluation methods show that 10 groups of video mosaic images have good mosaic effect. The research results provide the possibility for the high-quality implementation of panoramic photos and video stitching.

Panoramic video in education: A systematic literature review from 2011 to 2021

AbstractBackgroundPanoramic video as a novel instructional media with increasing educational application. However, empirical research on panoramic video instruction is mostly episodic with conditional and inconclusive results.ObjectivesThis study reviewed 10 years (2011–2021) of research on panoramic video in education to synthesize, meta‐analyse, and critically evaluate the state‐of‐art research findings, which can inform future educational practice and research directions.MethodsFollowing the protocol of systematic review, this study selected 83 articles from four major research databases and analysed them from five interrelated aspects: instructional context, technological features, instructional design, research design, and instructional effect. A meta‐analysis was also conducted to determine the overall effectiveness of panoramic video on education and the moderating factors.Results and ConclusionsThe results revealed a rapid upward trend in publication since 2018 with unequal distribution among regions and learning domains. Despite the technical advancement in immersion and interaction, panoramic videos were found to mainly support the acquisition of content and factual knowledge through direct instruction, and lack in‐video scaffolding and assessment. Moreover, many interventions were one‐time, short‐term studies that are susceptible of novelty effect. Lastly, an overall medium effect (g = 0.711) of panoramic video was reported and six significant moderating factors were identified based on the meta‐analysis results.Major takeaways and implicationsPanoramic videos are ubiquitously effective across grade levels, learning domains, and pedagogies, but are more suitable for teaching conceptual knowledge. Design features such as interaction, scaffolding, assessment, iteration, and video length deserve our special attention in panoramic video instruction.

Viewing Behavior Supported Visual Saliency Predictor for 360 Degree Videos

In virtual reality (VR), correct and precise estimations of user’s visual fixations and head movements can enhance the quality of experience by allocating more computation resources for analysing and rendering on the areas of interest. However, there is insufficient research about understanding the visual exploration of users when modeling VR visual attention. To bridge the gap between the saliency prediction for traditional 2D content and omnidirectional content, we construct the visual attention dataset and propose the visual saliency prediction framework for panoramic videos. Around the instantaneous viewing behavior, we propose a traditional method to adapt 2D saliency models and design a CNN-based model to better predict visual saliency. In the proposed traditional model, mechanism of visual attention and viewing behaviors are considered in the computation of edge weights on graphs which are interpreted as Markov chains. The fraction of the visual attention that is diverted to each high-clarity vision (HCV) area is estimated through equilibrium distribution of this chain. We also propose the Graph-Based CNN model. The RGB channel and optical flow form the spatial-temporal units of HCVs, from which node feature vectors are extracted. Graph convolution is used to learn the mutual information between node feature vectors of HCVs and retain geometric information. Then feature vectors are aligned according to geometry structure of equirectangular format, and the feature decoder maps the aligned feature maps to the data distribution. We also construct the dynamic omnidirectional monocular (DOM) saliency dataset with 64 diverse videos evaluated by 28 people. The subjective results show that the instantaneous viewing behavior is important in the VR experience. Extensive experiments are conducted on the dataset and the results demonstrate the effectiveness of the proposed framework. The dataset will be released to facilitate the future studies related to visual saliency prediction for 360-degree contents.

Research on the Application of Computer Big Data Technology in the Visual Communication of Sports Tennis Live Broadcast

This article introduces the realization of visual communication of sports tennis live broadcast system based on video splicing technology. First, the image is pre-processed, and then the SURF algorithm is used for image registration and splicing of the video image and the spliced image is merged. This article proposes a method for determining hot spots in panoramic videos based on motion foreground extraction. This method can smoothly track hot spots for extraction to achieve real-time sports tennis live broadcast. The implementation of this system saves manpower and financial resources in the process of live video streaming.

A new improved projection algorithms of virtual reality panoramic video and application in physical education

SUMMARYData storage and transmission have become the two biggest challenges it currently faces. In our study, we proposed an improved octahedral projection layout method. Performing an equivalent process on every patch to rework into an isosceles right‐angled triangle, combine into varied squares based on varied needs, and finally combine into a completely continuous interior and a facet magnitude relation of 2:1 flat parallelogram. Furthermore, the area‐preserving uniform projection method is also proposed. Moreover, applying virtual reality technology to Tai Chi teaching using research methods such as literature, expert interviews, questionnaires, teaching experiments, and mathematical statistics, the data of the basic physical quality test of the students in the experimental group and the control group were compared and analyzed to verify the scientific hypothesis that the VR technology could improve the teaching effect of Tai Chi. Results show that different from the compact layout of the initial octahedral projection, the approach proposed in our study has removed the inner discontinuous borderlines and possessed top boundary continuity. Students gain a good situational experience while deepening their understanding and memory of the details of Tai Chi movements, and the learning process is interesting and impressive.

Real-Time Multitarget Tracking for Panoramic Video Based on Dual Neural Networks for Multisensor Information Fusion

A multitarget real-time tracking system for panoramic video with multisensor information fusion dual neural networks is studied and implemented by combining dual neural networks, fused geometric features, and deep learning-based target real-time tracking algorithms. The motion model of multisensor information fusion is analyzed, the dual neural network perturbation model is introduced, and the state variables of the system are determined. Combined with the data structure model of multisensor information fusion, a least-squares-based optimization function is constructed. In this optimization function, the image sensor is constructed as an observationally constrained residual term based on the pixel plane, the inertial measurement unit is constructed as an observationally constrained residual term based on the motion model, and the motion relationship between sensors is constructed as a motion-constrained residual term to derive a mathematical model for real-time multitarget tracking of panoramic video. The simulation experimental results show that the integrated position estimation accuracy of the multisensor information fusion dual neural network-based panoramic video multitarget real-time tracking fusion algorithm is 10.5% higher than that of the optimal distributed panoramic video multitarget fusion algorithm without feedback. The results of the simulation experiments show that the integrated position estimation accuracy of the multitarget real-time tracking fusion algorithm for panoramic video improves by 10.96%, and the integrated speed estimation accuracy improves by 6.32% compared with the optimal distributed multitarget fusion algorithm without feedback. The proposed panoramic video multitarget real-time tracking algorithm based on the dual neural network can effectively improve the target tracking accuracy of the model on degraded frames (motion blur, target occlusion, out-of-focus, etc.), and the stability of the algorithm for target location and category detection is effectively improved by multiframe feature fusion. The research in this paper provides better technical support and research theory for panoramic video multitarget real-time tracking.

Tile-Based Panoramic Video Quality Assessment

As part of video transmission, video quality assessment plays an important role in improving the user’s visual quality and reducing transmission bitrate. However, the existing assessment schemes for the tile-based transmission mechanism cannot accurately predict the video quality. In this paper, we propose a tile-based panoramic video quality assessment method to optimize the transmission strategy for tile-based streaming. Considering the distortion of panoramic video projection, the inverse of gnomonic projection and bilinear interpolation are utilized to convert the planar tiles to sphere. Video Multimethod Assessment Fusion (VMAF) is applied to obtain the tile quality score. Besides, the user’s head movement and eye movement are combined to generate a tile weight map, and generate the final score of the panoramic video through weighted averaging. In addition, considering the impact of the quality fluctuation of the tiles within field of view, the tile quality loss model is established from multiple dimensions including the encoding parameter, viewport position and video content. Experimental results show that there is a high correlation between the predicted results and actual subjective scores, under two video quality assessment datasets. Furthermore, we integrate the method into the panoramic video transmission system. Compared with the tile-based adaptive rate algorithm, the proposed method degrades the video quality by only 3%, while reducing the bandwidth consumption by 16.99%.

Construction and evaluation of a 360 degrees panoramic video on the physical examination of nursing students

AimThe aim of this study was to construct and analyze the content validity and usability for a 360-degree panoramic video for physical examination. BackgroundAcquiring techniques and ability on physical examination (PE) skill is often difficult. Immersion and simulation provide an opportunity for nursing students in the real situation focus on the PE techniques practice and to improve patient care. The combination of digital instructional materials and innovative teaching strategies is crucial to improving education. Therefore, effective and clear instruction of PE techniques that can enable students confidently to use their skills in clinical practice is an issue that nursing education must address. MethodsThis study used a descriptive research design and the technology acceptance model to create a 360-degree video to teach students to perform PE. Content validity and usability were analyzed by five nursing teachers and 53 nursing students who had completed a physical assessment course were invited to assess its usability. ResultsThe results of the expert review indicated that the content validity of the video scored 0.98 in terms of content relevance, 0.95 in terms of convenience and 0.97 in terms of suitability, suggesting that the video was useful and appropriate for self-practicing. The experts’ qualitative feedback indicated that the technical demonstrations in the video were clear and specific and that the video was easy to use. The video increased learning motivation and streamlined learning. The ability to change the video’s perspective also increased learning interest. The results of a satisfaction survey were 8.94 points for perceived ease of use, 8.79 points for perceived usefulness, 8.56 points for user attitude toward the video and 9.02 points for continued intention to use the video, indicating that the participants were satisfied with the video. The participants’ qualitative feedback indicated that each PE technique was clear and that the ability to change the video’s perspective and zooming was useful and novel. ConclusionsThe 360-degree panoramic videos were described as a positive learning method for physical examination of nursing students. It provides easy to use and useful to practice PE technique under realistic conditions. Among these outcomes, the 360-degree video can be learned best when teaching PE in nursing education. Accordingly, nursing educators should construct similar videos for the PE of other parts of the body as soon as possible, improve the videos convenience and usefulness and promote their ability to strengthen professional skills.

Quad‐fisheye Image Stitching for Monoscopic Panorama Reconstruction

AbstractMonoscopic panorama provides the display of omnidirectional contents surrounding the viewer. An increasingly popular way to reconstruct a panorama is to stitch a collection of fisheye images. However, such non‐planar views may result in problems such as distortions and boundary irregularities. In most cases, the computational expense for stitching non‐planar images is also too high to satisfy real‐time applications. In this paper, a novel monoscopic panorama reconstruction pipeline that produces better quad‐fisheye image stitching results for omnidirectional environment viewing is proposed. The main idea is to apply mesh deformation for image alignment. To optimize inter‐lens parallaxes, unwarped images are firstly cropped and reshuffled to facilitate the circular environment scene composition by the seamless ring‐connection of the panorama borders. Several mesh constraints are then adopted to ensure a high alignment accuracy. After alignment, the boundary of the result is rectified to be rectangular to prevent gapping artefacts. We further extend our approach to video stitching. The temporal smoothness model is added to prevent unexpected artefacts in the panoramic videos. To support interactive applications, our stitching algorithm is programmed using CUDA. The camera motion and average gradient per video frame are further calculated to accelerate for synchronous real‐life panoramic scene reconstruction and visualization. Experimental results demonstrate that our method has advantages in respects of alignment accuracy, adaptability and image quality of the stitching result.

Multiple HD Screen-Based Virtual Studio System with Learned Mask-Free Portrait Harmonization

Virtual studio technology allows producers to combine live-action footage and computer-generated imagery and has promoted the development of film and television industry. However, the existing green screen-based virtual studio limits the visual effects just in postproduction. In this paper, we propose a new virtual studio system based on the recent emerging technology using multiple high-definition (HD) screens. Besides using traditional computer graphics like other similar systems, our system enables to capture panoramic video from the real world as the background and project it onto multiple HD screens, so as to recreate the scene in the studio. In addition, we propose a mask-free portrait harmonization network to make sure that the appearances of the foreground region and the background are consistent. Our portrait harmonization network does not need any auxiliary foreground mask and works in an end-to-end manner, which meets the postprocessing requirements for our virtual studio. Experimental result on real composite images shows that our network is superior to the state-of-the-art image harmonization method under the task of portrait harmonization. We also demonstrate our system with a CAVE system.

A Transcoding-Enabled 360° VR Video Caching and Delivery Framework for Edge-Enhanced Next-Generation Wireless Networks

Virtual reality (VR) content, including 360° panoramic video, provides users with an immersive multimedia experience and therefore attracts increasing research and development attention. However, the requirement of high bandwidth and low latency of virtual reality service demand puts forward greater challenges to the current infrastructure, especially mobile networks. Inspired by the sharable nature of virtual reality content tiles, we further considered the potential opportunities for computing, caching, and multicast to address the challenges of transmission of panoramic content. This paper proposes a novel transcoding-enabled VR video caching and delivery framework for edge-enhanced next-generation wireless networks. Firstly, an edge cooperative caching scheme based on multi-agent reinforcement learning is introduced to improve the utilization efficiency of computing and storage resources, and then reduce service delay. Second, a two-tier NOMA-based base station-multicast group matching mechanism is designed to solve the collaboration challenge during the edge delivery process. A series of experiments have demonstrated the advantages of the proposed scheme in terms of cache hit rate, latency and other aspects in comparison with alternative approaches.

Omnidirectional Panoramic Video System With Frame-by-Frame Ultrafast Viewpoint Control

This study developsan omnidirectional panoramic video system that can simultaneously observe zoom-in images at dozens of different viewpoints in a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> field of view by synchronizing hundreds of frames per second video shooting, with a rotation of tens-of-rotation-per-second mirror-rotator. When video shooting a scene with a certain exposure utilizing a mirror-rotator, apparent motion in the captured images becomes too large to observe non-blurred images, owing to its high-speed rotation. To reduce motion blurs caused by the rotation of a mirror-rotator, our system introduces an ultrafast pan-tilt mirror mechanism that can cancel apparent motion in images by executing frame-by-frame viewpoint control, so that the apparent speed in images becomes zero when the camera shutter is open, and it returns to its home position when the shutter is closed. Our system can capture 40 omnidirectionally different view images at 10 fps in real time by synchronizing 400-fps video shooting with frame-by-frame ultrafast pan-tilt mirror control for motion blur reduction when a mirror-rotator rotates at 10 rps. Its performance is verified by experimental results for several omnidirectional real scenes, and our system can capture clear and bright panoramic zoom-in images without decreasing the camera exposure when omnidirectionally observing dozens-of-meters-distance outdoor scenes.