Omnidirectional Camera Research Articles

FAST-FUSION: An Improved Accuracy Omnidirectional Visual Odometry System with Sensor Fusion and GPU Optimization for Embedded Low Cost Hardware

The main task while developing a mobile robot is to achieve accurate and robust navigation in a given environment. To achieve such a goal, the ability of the robot to localize itself is crucial. In outdoor, namely agricultural environments, this task becomes a real challenge because odometry is not always usable and global navigation satellite systems (GNSS) signals are blocked or significantly degraded. To answer this challenge, this work presents a solution for outdoor localization based on an omnidirectional visual odometry technique fused with a gyroscope and a low cost planar light detection and ranging (LIDAR), that is optimized to run in a low cost graphical processing unit (GPU). This solution, named FAST-FUSION, proposes to the scientific community three core contributions. The first contribution is an extension to the state-of-the-art monocular visual odometry (Libviso2) to work with omnidirectional cameras and single axis gyro to increase the system accuracy. The second contribution, it is an algorithm that considers low cost LIDAR data to estimate the motion scale and solve the limitations of monocular visual odometer systems. Finally, we propose an heterogeneous computing optimization that considers a Raspberry Pi GPU to improve the visual odometry runtime performance in low cost platforms. To test and evaluate FAST-FUSION, we created three open-source datasets in an outdoor environment. Results shows that FAST-FUSION is acceptable to run in real-time in low cost hardware and that outperforms the original Libviso2 approach in terms of time performance and motion estimation accuracy.

Using Music to Develop a Multisensory Communicative Environment for People with Late-Stage Dementia.

Background and ObjectivesResearch has indicated the benefit of music interventions on biological, psychological, and cognitive aspects of dementias, yet there is limited research focusing on music’s role in communication. This study developed a conceptual understanding of how people with late-stage dementia may express themselves nonverbally and interact with others during a live music group over time.Research Design and MethodsEight people with advanced dementias in residential care (aged 82–97 years), four care staff, and three musicians participated in 8-hr-long weekly live Music for Life sessions and listened to 1-hr-long recorded music session. Visual grounded theory was used to analyze video data collected nonintrusively via the Fly 360-degree camera.ResultsThe live music group facilitated a multisensory communicative environment allowing for verbal and nonverbal communicative actions, social interactional components and agency to develop over time. These aspects were influenced by three factors: time, one-to-one interaction within a group setting and the characteristics of the music.Discussion and ImplicationsNonverbal communication in later-stage dementia may be overlooked or underestimated by busy care staff and families. Using music as an interactive way to communicate can help develop mirroring and turn-taking which has been shown to improve quality of life for people with communication impairment, increase their nonverbal communication and allow for a connection to be built between people. Although further research is recommended, individuals responsible for residential care should feel confident that the development of ongoing music groups for this population is warranted as part of ongoing care.

OIDC-Net: Omnidirectional Image Distortion Correction via Coarse-to-Fine Region Attention

Omnidirectional cameras have recently received significant attention in panoramic imaging systems such as virtual reality (VR) technology; however, the strong geometric distortion in omnidirectional images severely affects the object recognition and semantic understanding. In this paper, we propose an automatic omnidirectional image distortion correction approach powered by a unified learning model (OIDC-Net). This approach is applicable for almost all types of omnidirectional cameras, requiring nothing more than a distorted image. A crucial and challenging ingredient for reconstructing the real physical scene is to estimate the heterogeneous distortion coefficients in an appropriate camera model. To address this issue, we present a novel coarse-to-fine region attention mechanism to alleviate the difficulty of predicting all coefficients simultaneously. With the proposed cascade structure and deep fusion strategy, the ambiguous relationship among these heterogeneous distortion coefficients has been incrementally perceived. Our experimental results show significant improvement over the state-of-the-art methods in terms of visual appearance, while maintaining a promising quantitative performance.

A FEASIBILITY STUDY ON INCREMENTAL BUNDLE ADJUSTMENT WITH FISHEYE IMAGES AND LOW-COST SENSORS

Abstract. Low cost imaging and positioning sensors are opening new frontiers for applications in near real-time Photogrammetry. Omnidirectional cameras acquiring images with 360° coverage, when combined with information coming from GNSS (Global Navigation Satellite Systems) and IMU (Inertial Measurement Unit), can efficiently estimate orientation and object space structure. However, several challenges remain in the use of low-cost sensors and image observations acquired by sensors with non-perspective inner geometry. The accuracy of the measurement using low-cost sensors is affected by different sources of errors and sensor stability. Microelectromechanical systems (MEMS) present a large gap between predicted and actual accuracy. This work presents a study on the performance of an integrated sensor orientation approach to estimate sensor orientation and 3D sparse point cloud, using an incremental bundle adjustment strategy and data coming from a low-cost portable mobile terrestrial system composed by off-theshelf navigation systems and a poly-dioptric system (Ricoh Theta S). Experiments were performed in an outdoor area (sidewalk), achieving a trajectory positional accuracy of 0.33 m and a meter level 3D reconstruction.

Synchronized Tracking in Multiple Omnidirectional Cameras with Overlapping View

Synchronized Tracking in Multiple Omnidirectional Cameras with Overlapping View

Real-Time Fall Detection Using Uncalibrated Fisheye Cameras

In this article, we describe an approach for the problem of fall detection among the several other everyday activities in indoor environment, using three uncalibrated fisheye cameras. The proposed methodology requires the input segmented silhouettes from the three simultaneously acquired frames, and it is capable of detecting fall events in every location of the imaged environment. The presented algorithm uses the model of fisheye image formation that is based on the spherical projection followed by central projection. Under this model, vertical structures are imaged as straight lines passing through the center of field of view, by a camera with approximately vertical optical axis. The main advantages of this article are the simplicity of the detecting rule, the speed of execution, and the utilization of heterogeneous omnidirectional cameras that allows simultaneous imaging along any direction. The proposed algorithm is designed and parameterized using an extensive data set of synthetic frames. The results from the real videos are presented using the frame statistics and the event-based statistics. The proposed algorithm correctly detects the fall events within standing or walking, as well as other nonfalling activities.

End-To-End Controls Using K-Means Algorithm for 360-Degree Video Control Method on Omnidirectional Camera-Equipped Autonomous Micro Unmanned Aircraft Systems

Micro unmanned aircraft systems (micro UAS)-related technical research is important because micro UAS has the advantage of being able to perform missions remotely. When an omnidirectional camera is mounted, it captures all surrounding areas of the micro UAS. Normal field of view (NFoV) refers to a view presented as an image to a user in a 360-degree video. The 360-degree video is controlled using an end-to-end controls method to automatically provide the user with NFoVs without the user controlling the 360-degree video. When using the end-to-end controls method that controls 360-degree video, if there are various signals that control the 360-degree video, the training of the deep learning model requires a considerable amount of training data. Therefore, there is a need for a method of autonomously determining the signals to reduce the number of signals for controlling the 360-degree video. This paper proposes a method to autonomously determine the output to be used for end-to-end control-based deep learning model to control 360-degree video for micro UAS controllers. The output of the deep learning model to control 360-degree video is automatically determined using the K-means algorithm. Using a trained deep learning model, the user is presented with NFoVs in a 360-degree video. The proposed method was experimentally verified by providing NFoVs wherein the signals that control the 360-degree video were set by the proposed method and by user definition. The results of training the convolution neural network (CNN) model using the signals to provide NFoVs were compared, and the proposed method provided NFoVs similar to NFoVs of existing user with 24.4% more similarity compared to a user-defined approach.

Accurate and Robust Monocular SLAM with Omnidirectional Cameras.

Simultaneous localization and mapping (SLAM) are fundamental elements for many emerging technologies, such as autonomous driving and augmented reality. For this paper, to get more information, we developed an improved monocular visual SLAM system by using omnidirectional cameras. Our method extends the ORB-SLAM framework with the enhanced unified camera model as a projection function, which can be applied to catadioptric systems and wide-angle fisheye cameras with 195 degrees field-of-view. The proposed system can use the full area of the images even with strong distortion. For omnidirectional cameras, a map initialization method is proposed. We analytically derive the Jacobian matrices of the reprojection errors with respect to the camera pose and 3D position of points. The proposed SLAM has been extensively tested in real-world datasets. The results show positioning error is less than 0.1% in a small indoor environment and is less than 1.5% in a large environment. The results demonstrate that our method is real-time, and increases its accuracy and robustness over the normal systems based on the pinhole model.

Automatic Reconstruction of Building-Scale Indoor 3D Environment with a Deep-Reinforcement-Learning-Based Mobile Robot

The aim of this paper is to digitize the environments in which humans live, at low cost, and reconstruct highly accurate three-dimensional environments that are based on those in the real world. This three-dimensional content can be used such as for virtual reality environments and three-dimensional maps for automatic driving systems. In general, however, a three-dimensional environment must be carefully reconstructed by manually moving the sensors used to first scan the real environment on which the three-dimensional one is based. This is done so that every corner of an entire area can be measured, but time and costs increase as the area expands. Therefore, a system that creates three-dimensional content that is based on real-world large-scale buildings at low cost is proposed. This involves automatically scanning the indoors with a mobile robot that uses low-cost sensors and generating 3D point clouds. When the robot reaches an appropriate measurement position, it collects the three-dimensional data of shapes observable from that position by using a 3D sensor and 360-degree panoramic camera. The problem of determining an appropriate measurement position is called the “next best view problem,” and it is difficult to solve in a complicated indoor environment. To deal with this problem, a deep reinforcement learning method is employed. It combines reinforcement learning, with which an autonomous agent learns strategies for selecting behavior, and deep learning done using a neural network. As a result, 3D point cloud data can be generated with better quality than the conventional rule-based approach.

Understanding relationships between perceived greenery and visible green ratio measured from omnidirectional camera images

Understanding relationships between perceived greenery and visible green ratio measured from omnidirectional camera images

Comparison of Tracking Techniques on 360-Degree Videos

With the availability of 360-degree cameras, 360-degree videos have become popular recently. To attach a virtual tag on a physical object in 360-degree videos for augmented reality applications, automatic object tracking is required so the virtual tag can follow its corresponding physical object in 360-degree videos. Relative to ordinary videos, 360-degree videos in an equirectangular format have special characteristics such as viewpoint change, occlusion, deformation, lighting change, scale change, and camera shakiness. Tracking algorithms designed for ordinary videos may not work well on 360-degree videos. Therefore, we thoroughly evaluate the performance of eight modern trackers in terms of accuracy and speed on 360-degree videos. The pros and cons of these trackers on 360-degree videos are discussed. Possible improvements to adapt these trackers to 360-degree videos are also suggested. Finally, we provide a dataset containing nine 360-degree videos with ground truth of target positions as a benchmark for future research.

DETEKSI DAN PREDIKSI TRAJEKTORI OBJEK BERGERAK DENGAN OMNI-VISION MENGGUNAKAN PSO-NN DAN INTERPOLASI POLYNOMIAL

In the Indonesian wheeled soccer robot competition in one team consists of three robots, where one robot is a goalkeeper. In the competition the movement of robots and balls is very dynamic. So that a method is needed to predict the movement of the ball so that the goalkeeper can anticipate the movement of the ball. In this research the ball detected by digital image processing and Particle Swarm Optimization-Neural Network (PSO-NN) is used as a calibration model for object distance through omnidirectional cameras. The interpolation approach of the polynomial curve is used to obtain estimates of the model from two-dimensional data from detected objects. The results showed that the distance conversion in object detection with the PSO-NN model obtained 0.13% in percentage of average squared error (PMSE) measurement and 20% in an average prediction error.

A system for monitoring farmed fish via LED‐based visible light communication

We propose a novel fish farm monitoring system utilizing visible light communication technology. In our system, a sensor node equipped with a full‐color LED module is attached to fish to transmit sensor data. A sink node equipped with an omnidirectional camera at the bottom of the cage receives sensor data. This is a simple, low‐energy approach for data transmission. We establish the fundamental performance of our proposed system through simulations. © 2019 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.

Obstacle Avoidance using Fuzzy Logic Controller on Wheeled Soccer Robot

The purpose of this study is to apply Fuzzy Logic Controller on a wheeled soccer robot to avoid the collision with other robots in the field. The robot equipped by an omnidirectional camera as a vision sensor, a mini-PC for the image processing device, a microcontroller to handle I/O system, and three wheel's omnidirectional mover system. Omni-camera produces four input-values, namely: X coordinate ball position, Y coordinate ball position, distance and angle from obstacle to the point of interest in the camera frame. These inputs processed by a mini-PC and then forward to a microcontroller to calculate the output using Fuzzy Logic Controller. The output variables are the movement rate of the robot in the X, and Y coordinate. These outputs will be used by the kinematics controller to manage the speed of three Omni-wheels driven by 24 volts DC motors. The experiment shows a good result with the percentage of the success of the robot catching the ball is around 70% and 80% in avoiding the obstacle. In time performance, the soccer robot with Fuzzy Logic Controller is superior by 4.67 seconds compared to the robot without this method.

Unpacking the Potential of 360degree Video to Support Pre-Service Teacher Development

AbstractVideo has become a useful tool in Initial Teacher Education for self-evaluation and reflection by pre-service teachers (PSTs). The availability of 360degree video cameras and web-based applications, to review and annotate 360degree videos, allows PSTs greater flexibility to view and review their practice from a variety of perspectives. This study explores PSTs’ use of 360degree video for reflection on their teaching practice. 360degree video provides PSTs with the capacity to pan ‘around’ the video, and in doing so has supported PSTs to detach and reflect on their own practice. The findings suggest that the PSTs valued the additional perspectives afforded by the 360degree nature of the video, which had a particular impact on their understanding of their presence, interactions and explanations. Peer video viewing was also found to be a useful tool in supporting PSTs to ‘notice’ additional areas for improvement in their own practice.

Effectiveness of seismic strengthening to repeated earthquakes in historic urban contexts

PurposeThe purpose of this paper is to discuss the dynamics of the evolution of damage to the residential buildings within the city walls of Norcia during the six-month seismic swarm that hit Central Italy in the period 24th August 2016 to end of January 2019. This is accomplished by comparing the damage recorded by the Italian Civil Protection usability form (AEDES form) during this period after each event.Design/methodology/approachFirst, these outputs are compared with a qualitative assessment conducted by means of omnidirectional camera (ODC) imagery collected on site by the authors, to explore the ability of this technology to support post-earthquake damage assessment. The damage level attributed with these two techniques is then further compared with the output of the analytical vulnerability assessment method FaMIVE, which allows to correlate damage to vulnerability. Specifically, the objective is to investigate the efficacy and performance of historic and recent strengthening interventions.FindingsResults show that there is a good correspondence between AeDES and ODC assessments for low to medium damage grades (DG). Discrepancies in higher DGs are discussed in light of the different levels of information that can be recorded by using the two tools. The efficacy of strengthening is also well captured by the FaMIVE method. The procedure estimates a decrease of almost 40, 25 and 20 per cent of the total number of buildings failing out-of-plane, respectively, for the three seismic events considered, when restraining elements are in action.Research limitations/implicationsThe analysis conducted in this work make use of deterministic values of Norcia’s masonry fabric characteristics that have been found in literature, thus implying that neither the probabilistic aspects nor the related uncertainties have been properly investigated and addressed. However, this limitation is to be considered within the more general context of the legislation for the preservation of historic buildings which limits substantially any type of semi-destructive tests, hence limiting the reliability of the values available in literature. This in turn affects the decisions informing the design and implementation of strengthening interventions which can be confidently considered reliable and effective.Originality/valueThe paper addresses for the first time a systematic investigation of damage progression in historic masonry structures, part of urban aggregates in heritage cities. The current urban fabric is discussed in view of historic building codes as the basis for determining the present seismic vulnerability of the historic city centre of Norcia. The study provides new data sets for the city of Norcia and develops a statistical correlation between cumulative damage and analytical vulnerability functions for heritage buildings exposed to a swarm of earthquakes. The analytical assessment of the effect of historic strengthening is totally novel.

Scalable 360° Video Stream Delivery: Challenges, Solutions, and Opportunities

In recent years, virtual reality and augmented reality applications have seen a significant increase in popularity. This is due to multiple technology trends. First, the availability of new tethered and wireless head-mounted displays allows viewers to consume new types of content. Second, 360° omnidirectional cameras, in combination with production software, make it easier to produce personalized 360° videos. Third, beyond these new developments for creating and consuming such content, video sharing websites and social media platforms enable users to publish and view 360° video content. In this paper, we present challenges of 360° video streaming systems, give an overview of existing approaches for 360° video streaming, and outline research opportunities enabled by 360° video. We focus on the data model for 360° video and the different challenges and approaches of creating, distributing, and presenting 360° video content, including 360° video recording, storage, distribution, edge delivery, and quality-of-experience evaluation. In addition, we identify major research opportunities with respect to efficient storage, timely distribution, and cybersickness-free personalized viewing of 360° videos.

Affordable person detection in omnidirectional cameras using radial integral channel features

Omnidirectional cameras cover more ground than perspective cameras, at the expense of resolution. Their comprehensive field of view makes omnidirectional cameras appealing for security and ambient intelligence applications. Person detection is usually a core part of such applications. Conventional methods fail for omnidirectional images due to different image geometry and formation. In this study, we propose a method for person detection in omnidirectional images, which is based on the integral channel features approach. Features are extracted from various channels, such as LUV and gradient magnitude, and classified using boosted decision trees. Features are pixel sums inside annular sectors (doughnut slice shapes) contained by the detection window. We also propose a novel data structure called radial integral image that allows to calculate sums inside annular sectors efficiently. We have shown with experiments that our method outperforms the previous state of the art and uses significantly less computational resources.

A Comparison of Adaptive View Techniques for Exploratory 3D Drone Teleoperation

Drone navigation in complex environments poses many problems to teleoperators. Especially in three dimensional (3D) structures such as buildings or tunnels, viewpoints are often limited to the drone’s current camera view, nearby objects can be collision hazards, and frequent occlusion can hinder accurate manipulation. To address these issues, we have developed a novel interface for teleoperation that provides a user with environment-adaptive viewpoints that are automatically configured to improve safety and provide smooth operation. This real-time adaptive viewpoint system takes robot position, orientation, and 3D point-cloud information into account to modify the user’s viewpoint to maximize visibility. Our prototype uses simultaneous localization and mapping (SLAM) based reconstruction with an omnidirectional camera, and we use the resulting models as well as simulations in a series of preliminary experiments testing navigation of various structures. Results suggest that automatic viewpoint generation can outperform first- and third-person view interfaces for virtual teleoperators in terms of ease of control and accuracy of robot operation.

Reflectance and Shape Estimation with a Light Field Camera Under Natural Illumination

Reflectance and shape are two important components in visually perceiving the real world. Inferring the reflectance and shape of an object through cameras is a fundamental research topic in the field of computer vision. While three-dimensional shape recovery is pervasive with varieties of approaches and practical applications, reflectance recovery has only emerged recently. Reflectance recovery is a challenging task that is usually conducted in controlled environments, such as a laboratory environment with a special apparatus. However, it is desirable that the reflectance be recovered in the field with a handy camera so that reflectance can be jointly recovered with the shape. To that end, we present a solution that simultaneously recovers the reflectance and shape (i.e., dense depth and normal maps) of an object under natural illumination with commercially available handy cameras. We employ a light field camera to capture one light field image of the object, and a 360-degree camera to capture the illumination. The proposed method provides positive results in both simulation and real-world experiments.