Omnidirectional Camera Research Articles

Time Delay Mitigation in Aerial Telerobotic Operations Using Heterogeneous Stereo-Vision Systems

This paper investigates the use of a heterogeneous stereo-vision system to mitigate the effects of time delays in a drone-based visual interface presented to a human operator. Time delays in the display for a telerobotic interface refer to the time difference between the operator’s input action and the corresponding visible outcome. In human/machine interfaces, time delays can arise due to computation, telecommunication, and mechanical limitations. These delays can degrade the performance of the human/machine system. A heterogeneous stereo-vision predictive algorithm is presented that can reduce the negative effects of time delays in the operator’s display. The heterogeneous stereo-vision system consists of an omnidirectional camera and a pan/tilt/zoom camera. Two predictive display setups were developed that modify the delayed video imagery that would otherwise be presented to the operator in a way that provides an almost immediate visual response to the operator’s control actions. The usability of the system is determined through human performance testing with and without the predictive algorithms. The results indicate that the predictive algorithm allows more efficient, accurate, and user-friendly operation.

Using Virtual Reality Head-Mounted Displays to Assess Skills in Emergency Medicine: Validity Study.

Many junior doctors must prepare to manage acutely ill patients in the emergency department. The setting is often stressful, and urgent treatment decisions are needed. Overlooking symptoms and making wrong choices may lead to substantial patient morbidity or death, and it is essential to ensure that junior doctors are competent. Virtual reality (VR) software can provide standardized and unbiased assessment, but solid validity evidence is necessary before implementation. This study aimed to gather validity evidence for using 360-degree VR videos with integrated multiple-choice questions (MCQs) to assess emergency medicine skills. Five full-scale emergency medicine scenarios were recorded with a 360-degree video camera, and MCQs were integrated into the scenarios to be played in a head-mounted display. We invited 3 groups of medical students with different experience levels to participate: first- to third-year medical students (novice group), last-year medical students without emergency medicine training (intermediate group), and last-year medical students with completed emergency medicine training (experienced group). Each participant's total test score was calculated based on the number of correct MCQ answers (maximum score of 28), and the groups' mean scores were compared. The participants rated their experienced presence in emergency scenarios using the Igroup Presence Questionnaire (IPQ) and their cognitive workload with the National Aeronautics and Space Administration Task Load Index (NASA-TLX). We included 61 medical students from December 2020 to December 2021. The experienced group had significantly higher mean scores than the intermediate group (23 vs 20; P=.04), and the intermediate group had significantly higher scores than the novice group (20 vs 14; P<.001). The contrasting groups' standard-setting method established a pass-or-fail score of 19 points (68% of the maximum possible score of 28). Interscenario reliability was high, with a Cronbach α of 0.82. The participants experienced the VR scenarios with a high degree of presence with an IPQ score of 5.83 (on a scale from 1-7), and the task was shown to be mentally demanding with a NASA-TLX score of 13.30 (on a scale from 1-21). This study provides validity evidence to support using 360-degree VR scenarios to assess emergency medicine skills. The students evaluated the VR experience as mentally demanding with a high degree of presence, suggesting that VR is a promising new technology for emergency medicine skills assessment.

Information presentation and gaze-following for grounding in simulated anaesthesia emergencies: A multimodal analysis

Unanticipated difficult intubation is a relatively common problem anaesthetists face in everyday practice. Algorithms to manage the problem were issued (Japanese Society of Anesthesiologists, 2014), but how anaesthetists actually perceive the problem and respond to the difficulties with other healthcare professionals present is still to be uncovered. Drawing on the concept of resilient healthcare (Hollnagel et al., 2013), which values “how things go well”, this study analysed interactions in simulated emergencies between a less experienced anaesthetist (trainee) and an experienced anaesthetist (trainer). The latter took a role as a nurse and led the scenario of difficult intubation. Five sessions with the same trainer, one of five different trainees and a manikin as a patient were recorded with a 360-degree camera in an operating room at a large teaching hospital in Japan. The data was transcribed and analysed with discourse and multimodal corpus analytic approaches. In the simulated interactions, four phases of grounding (Clark, 1996) were observed in the joint decision-making process between the trainer and the trainee: (1) the trainer’s information presentation with her gaze address at the referent, (2) the trainee’s gaze following and acknowledgement of information reception, (3) the trainer’s prompt for the trainee’s decision-making, and (4) the trainee’s (or trainer’s) giving instruction to manage the difficulty. The trainer presented the patient’s condition and visual/audio information affordable in the environment (e.g., signals on a vital monitor), using verbal and multimodal resources, i.e., gaze address and deictic gestures. The participants as individual sensory agencies were engaged in the embodied process of shared sense-making in the particular context to establish common ground for joint decision-making on immediate actions required.

Mutual visibility by fat robots with slim omnidirectional camera

In the existing literature of the Mutual Visibility problem for autonomous robot swarms, the adopted visibility models have some idealistic assumptions that are not consistent with practical sensing device implementations. This paper investigates the problem in the more realistic visibility model called opaque fat robots with slim omnidirectional camera. The robots are modeled as unit disks, each having an omnidirectional camera represented as a disk of smaller size. The region obstructed by a single robot in this model is a truncated infinite cone. This makes the visibility model significantly challenging compared to the previous ones. We assume that the robots have compasses that allow agreement in the direction and orientation of both axes of their local coordinate systems. The robots are equipped with visible lights which serve as a medium of communication and also as a form of memory. We present a distributed algorithm for the Mutual Visibility problem which is provably correct in the semi-synchronous setting. Our algorithm also provides a solution for Leader Election which we use as a subroutine in our main algorithm. Although Leader Election is trivial with two axis agreement in the full visibility model, it is challenging in our case and is of independent interest.

High-performance omnidirectional self-powered photodetector constructed by CsSnBr3/ITO heterostructure film

Omnidirectional photodetectors attract enormous attention due to their prominent roles in optical tracking systems and omnidirectional cameras. However, it is still a challenge for the construction of high-performance omnidirectional photodetectors where the incident light can be effectively absorbed in multiple directions and the photo-generated carriers can be effectively collected. Here, a high-performance omnidirectional self-powered photodetector based on the CsSnBr3/indium tin oxide (ITO) heterostructure film was designed and demonstrated. The as-fabricated photodetector exhibited excellent self-powered photodetection performance, showing responsivity and detectivity up to 35.1 ​mA/W and 1.82 ​× ​1010 Jones, respectively, along with the smart rise/decay response time of 4 ​ms/9 ​ms. Benefitting from the excellent photoelectric properties of the CsSnBr3 film as well as the ability of the CsSnBr3/ITO heterostructure to efficiently separate and collect photo-generated carriers, the as-fabricated photodetector also exhibited excellent omnidirectional self-powered photodetection performance. All the results have certified that this work finds an efficient way to realize high-performance omnidirectional self-powered photodetectors.

Review and analysis of the properties of 360-degree surround view cameras in autonomous vehicles

Autonomous vehicles are becoming increasingly prevalent in today's world, and the demand for efficient and safe self-driving technology has never been higher. One key component of this technology is the 360-degree surround view camera, which provides a complete view of the vehicle's surroundings, allowing it to navigate safely and efficiently. In this paper, we conduct a comprehensive review and analysis of the properties of these cameras in autonomous vehicles. We begin by exploring the various types of 360-degree cameras available, including fisheye, parabolic, and mirror-based designs. We then delve into the technical specifications of these cameras, including resolution, frame rate, and field of view. We also examine the various image processing techniques used to enhance the quality of the images captured by these cameras, such as image stabilization and noise reduction. Next, we discuss the benefits and drawbacks of using 360-degree surround view cameras in autonomous vehicles. On the one hand, these cameras offer a complete view of the vehicle's surroundings, allowing for safe and efficient navigation. On the other hand, they can be expensive and may not always provide the necessary level of detail for certain driving scenarios. We also discuss the various challenges that must be overcome when using 360-degree cameras in autonomous vehicles, such as occlusions, lighting conditions, and the need for real-time processing. Finally, we conclude by discussing the future of 360-degree camera technology in autonomous vehicles and the potential for further advancements in this field. Overall, this paper provides a comprehensive review and analysis of the properties of 360-degree surround view cameras in autonomous vehicles, offering insights into their benefits, drawbacks, and technical specifications.

Rapid and Effective Technology Development for 3D-Model-Based Solar Access Analysis and Comparative Study with Fish-Eye Camera

In this study, we developed a 3D-model-based technology that can evaluate solar access by analyzing solar radiation and shade to find the optimal location for a solar system. We developed an algorithm that can quickly calculate viewshed by applying ray-casting technology, which is useful in the field of computer graphics. To apply the developed technology, an unmanned aerial vehicle (DJI MAVIC 3) was used to create a 3D model by taking 320 photos of the Kangwon National University Samcheok campus. To verify the developed technology, a comparison with image-based analysis using a 360-degree camera was performed for 30 points. As a result of applying the developed technology to the study area, it was possible to calculate the solar access for each point. In general, image-based analysis exaggerates the effects of objects such as trees, whereas the developed technique can produce realistic results if the 3D objects were well built. If the technology is further developed in the future, it can be used to increase the efficiency of solar power generation.

Equirectangular Image Data Detection, Segmentation and Classification of Varying Sized Traffic Signs: A Comparison of Deep Learning Methods.

There are known limitations in mobile omnidirectional camera systems with an equirectangular projection in the wild, such as momentum-caused object distortion within images, partial occlusion and the effects of environmental settings. The localization, instance segmentation and classification of traffic signs from image data is of significant importance to applications such as Traffic Sign Detection and Recognition (TSDR) and Advanced Driver Assistance Systems (ADAS). Works show the efficacy of using state-of-the-art deep pixel-wise methods for this task yet rely on the input of classical landscape image data, automatic camera focus and collection in ideal weather settings, which does not accurately represent the application of technologies in the wild. We present a new processing pipeline for extracting objects within omnidirectional images in the wild, with included demonstration in a Traffic Sign Detection and Recognition (TDSR) system. We compare Mask RCNN, Cascade RCNN, and Hybrid Task Cascade (HTC) methods, while testing RsNeXt 101, Swin-S and HRNetV2p backbones, with transfer learning for localization and instance segmentation. The results from our multinomial classification experiment show that using our proposed pipeline, given that a traffic sign is detected, there is above a 95% chance that it is classified correctly between 12 classes despite the limitations mentioned. Our results on the projected images should provide a path to use omnidirectional images with image processing to enable the full surrounding awareness from one image source.

Coarse-to-Fine Hybrid 3D Mapping System With Co-Calibrated Omnidirectional Camera and Non-Repetitive LiDAR

This letter presents a novel 3D mapping robot with an omnidirectional field-of-view (FoV) sensor suite composed of a non-repetitive LiDAR and an omnidirectional camera. Thanks to the non-repetitive scanning nature of the LiDAR, an automatic targetless co-calibration method is proposed to simultaneously calibrate the intrinsic parameters for the omnidirectional camera and the extrinsic parameters for the camera and LiDAR, which is crucial for the required step in bringing color and texture information to the point clouds in surveying and mapping tasks. Comparisons and analyses are made to target-based intrinsic calibration and mutual information (MI)-based extrinsic calibration, respectively. With this co-calibrated sensor suite, the hybrid mapping robot integrates both the odometry-based mapping mode and stationary mapping mode. Meanwhile, we proposed a new workflow to achieve coarse-to-fine mapping, including efficient and coarse mapping in a global environment with odometry-based mapping mode; planning for viewpoints in the region-of-interest (ROI) based on the coarse map (relies on the previous work [1]); navigating to each viewpoint and performing finer and more precise stationary scanning and mapping of the ROI. The fine map is stitched with the global coarse map, which provides a more efficient and precise result than the conventional stationary approaches and the emerging odometry-based approaches, respectively.

Live360: Viewport-Aware Transmission Optimization in Live 360-Degree Video Streaming

Live 360-degree video streaming is emerging as the next disruptive innovation in the coming era. However, compared to conventional live video streaming, it is much more challenging to provide high-quality live 360-degree video streaming due to high bandwidth demand, stringent latency requirement, and diverse user viewports. In this paper, we propose a viewport-aware live 360-degree video streaming framework called Live360 to optimize end-to-end video stream transmission. To make the problem tractable, we decompose the transmission optimization problem into two sub-problems, and optimize upstream and downstream transmission in an asynchronous way. We prioritize 360-degree cameras according to users’ real-time viewing interests and upload attractive contents with a higher bitrate. We also re-define the metric of QoE (Quality-of-Experience) for live 360-degree video viewers, and solve the optimization problems efficiently using dynamic programming. Extensive experiments show that Live360 is lightweight and significantly outperforms other state-of-the-art methods in terms of various QoE metrics. The average QoE of Live360 is twice that of other baseline methods.

A monocular wide-field speed sensor inspired by the crabs’ visual system for traffic analysis

The development of visual sensors for traffic analysis can benefit from mimicking two fundamental aspects of the visual system of crabs: their panoramic vision and their visual processing strategy adapted to a flat world. First, the use of omnidirectional cameras in urban environments allows for analyzing the simultaneous movement of many objects of interest over broad areas. This would reduce the costs and complications associated with infrastructure: installation, synchronization, maintenance, and operation of traditional vision systems that use multiple cameras with a limited field of view. Second, in urban traffic analysis, the objects of interest (e.g. vehicles and pedestrians) move on the ground surface. This constraint allows the calculation of the 3D trajectory of the vehicles using a single camera without the need to use binocular vision techniques.The main contribution of this work is to show that the strategy used by crabs to visually analyze their habitat (monocular omnidirectional vision with the assumption of a flat world ) is useful for developing a simple and effective method to estimate the speed of vehicles on long trajectories in urban environments. It is shown that the proposed method estimates the speed with a root mean squared error of 2.7 km h−1.

Small-Scale Demonstration of Remote Control of Patrol and Work Robot with Arms Employing Local 5G System

Local (private) 5G system can provide a secure and flexible network using the cellular-based technologies at their facilities (e.g., factories, agricultural lands, and buildings). We constructed a small-scale demonstration system that exhibits the remote control of a patrol and work robot with arms using a local 5G system. The constructed robot comprises a robot operating system-based unmanned ground vehicle, two laser range finders, a webcam, an omnidirectional camera, and a six-axis robot arm. To fabricate a demonstration system with open-source software, we assessed the one-way delay of video streaming by changing different CPU, camera types, drivers, applications, and video resolutions. According to the assessment findings, it was demonstrated that it is possible to realize approximately 100ms delay under the limited resolution condition, and the allowable maximum absolute delay of 300ms can be attained even for full HD (1920 × 1080) resolution of this demonstration. Furthermore, local 5G was demonstrated to reduce delay variations to the same level as wired systems. It was also clarified that the increase in delay due to the application of local 5G is relatively small (5-25% in total delay) in this demonstration. Finally, we employed the small-scale demonstration system for the online and onsite campus tours for high school students.

Analysis of the relationship between nighttime illuminance and fear of crime using a quasi-controlled experiment with recorded virtual reality

By employing a quasi-controlled experiment with recorded virtual reality (VR), this study examines the impact of nighttime illuminance on fear of crime in four narrow streets without sidewalks in Seoul, South Korea. In April 2021, the horizontal and vertical illuminance of four streets was measured at 13 different time slots between 6 pm and 11 pm, and immediately afterward, the environment of the streets was recorded with a 360-degree camera. 101 young adult participants were asked to answer the level of fear of crime perceived in each video while experiencing 10 randomly selected VR videos out of 40 using head-mounted display. The results of the cross-classified multilevel analysis show that fear of crime can be reduced not only by enhancing average and minimum horizontal and vertical illuminance levels but also by improving the uniformity ratio of illuminance. In addition, the results are more evident in the video taken in the residential areas (Sites A and B) than in the commercial areas (Sites C and D). These results could be applied to nighttime illuminance management in narrow urban streets.

Rapid tunnel scanning using a 360-degree camera and SfM photogrammetry

Photogrammetric scanning can be employed for the digitization of underground spaces, for example for remote mapping, visualization, or training purposes. However, such a technique requires capturing many photos, which can be laborious and time-consuming. Previous research has demonstrated that the acquisition time can be reduced by capturing the data with multiple lenses or devices at the same time. Therefore, this paper demonstrates a method for rapid scanning of hard rock tunnels using Structure-from-Motion (SfM) photogrammetry and a 360-degree camera. The test was performed in the Underground Research Laboratory of Aalto University (URLA). The tunnel is located in granitic rocks at a depth of 20 m below the Otaniemi campus in Espoo, Finland. A 10 m long and 3.5 m high tunnel section with exposed rock was selected for this study. Photos were captured using the 360-degree camera from 27 locations and 3D models were reconstructed using SfM photogrammetry. The accuracy, speed, and resolution of the 3D models were measured and compared with models scanned with a digital single-lens reflex (DSLR) camera. The results show that the data capture process with a 360-degree camera is 6x faster compared to a conventional camera. In addition, the orientation of discontinuities was measured remotely from the 3D model and the digitally obtained values matched the manual compass measurements. Even though the quality of the 360-degree camerabased 3D model was visually inferior compared to the DSLR model, the point cloud had sufficient accuracy and resolution for semi-automatic discontinuity measurements. The quality of the models can be improved by combining 360-degree and DSLR photos which result in a point cloud with 3x higher resolution and 2x higher accuracy. The results demonstrated that 360-degree cameras can be used for the rapid digitization of underground tunnels.

Teacher-Student-Environment Interactions in Fieldwork Through 360-Degree Camera

The 360-degree camera is a relatively new technology that might become an effective tool for the development of students and teachers and overall educational improvement. As “normal” video provides a maximum 130-degree perspective, it could not offer a complex capture of education. This paper presents a way to use a 360-degree camera during two different modes of trainee teachers’ fieldwork and proposes how to evaluate this form of education in terms of teacher-student-environment interactions by identifying the occurrence and duration of each type of interaction. It is evident that the type and intensity of interactions affect fieldwork significantly and may lead to different depths of learning.

A Three-Dimensional Calibration and Reconstruction Method Using Omnidirectional Cameras with Control Points of Unknown Coordinates

A Three-Dimensional Calibration and Reconstruction Method Using Omnidirectional Cameras with Control Points of Unknown Coordinates

Using Omnidirectional Cameras to Measure Position of a Motorcycle

Using Omnidirectional Cameras to Measure Position of a Motorcycle

Position Measuring System for a Motorcycle Using Quad Omnidirectional Cameras

Position Measuring System for a Motorcycle Using Quad Omnidirectional Cameras

Research on pose calibration method for omnidirectional camera and rotation axis

Research on pose calibration method for omnidirectional camera and rotation axis

Lightweight Super-Resolution with Self-Calibrated Convolution for Panoramic Videos

Panoramic videos are shot by an omnidirectional camera or a collection of cameras, and can display a view in every direction. They can provide viewers with an immersive feeling. The study of super-resolution of panoramic videos has attracted much attention, and many methods have been proposed, especially deep learning-based methods. However, due to complex architectures of all the methods, they always result in a large number of hyperparameters. To address this issue, we propose the first lightweight super-resolution method with self-calibrated convolution for panoramic videos. A new deformable convolution module is designed first, with self-calibration convolution, which can learn more accurate offset and enhance feature alignment. Moreover, we present a new residual dense block for feature reconstruction, which can significantly reduce the parameters while maintaining performance. The performance of the proposed method is compared to those of the state-of-the-art methods, and is verified on the MiG panoramic video dataset.