Panoramic Camera Research Articles

A Pipeline for Automated Processing of Declassified Corona KH-4 (1962–1972) Stereo Imagery

The Corona KH-4 reconnaissance satellite missions from 1962-1972 acquired panoramic stereo imagery with high spatial resolution of 1.8-7.5 m. The potential of 800,000+ declassified Corona images has not been leveraged due to the complexities arising from handling of panoramic imaging geometry, film distortions and limited availability of the metadata required for georeferencing of the Corona imagery. This paper presents Corona Stereo Pipeline (CoSP): A pipeline for processing of Corona KH-4 stereo panoramic imagery. CoSP utlizes a deep learning based feature matcher SuperGlue to automatically match features point between Corona KH-4 images and recent satellite imagery to generate Ground Control Points (GCPs). To model the imaging geometry and the scanning motion of the panoramic KH-4 cameras, a rigorous camera model consisting of modified collinearity equations with time dependent exterior orientation parameters is employed. The results show that using the entire frame of the Corona image, bundle adjustment using well-distributed GCPs results in an average standard deviation (SD) of less than 2 pixels. The distortion pattern of image residuals of GCPs and y-parallax in epipolar resampled images suggest that film distortions due to long term storage as likely cause of systematic deviations. Compared to the SRTM DEM, the Corona DEM computed using CoSP achieved a Normalized Median Absolute Deviation (NMAD) of elevation differences of ~4 m over an area of approx. 4000 $km^2$. We show that the proposed pipeline can be applied to sequence of complex scenes involving high relief and glacierized terrain and that the resulting DEMs can be used to compute long term glacier elevation changes over large areas.

Dynamic SLAM of binocular catadioptric panoramic camera based on inverse depth filter

Dynamic SLAM of binocular catadioptric panoramic camera based on inverse depth filter

Aqua-Vision (An underwater panoramic camera with computer vision and analytics)

In the present social media-bound lifestyle, capturing memories and keeping them accessible is gaining a significant demand globally. For this purpose, a robust, portable camera system for recreational or commercial purposes can be of substantial advantage to society. Aqua-Vision intends to bring an affordable underwater camera system with various innovative features to the hands of consumers. The smart module consists of a waterproof gimbal camera that can be used underwater, providing a hassle-free and reliable user experience and offers rotary motion along two axes. The camera features various general modes like panorama, burst shot, and smart modes using inbuilt computer vision algorithms. The gimbal camera setup can be controlled and switched remotely between all possible modes using a mobile application. All the above features will allow the user to capture photos/videos in any possible setup and use the camera module for various applications. The advent of such innovative, convenient, and robust modules will help cater to the market demands effectively.

Panoramic Visual-Inertial SLAM Tightly Coupled with a Wheel Encoder

This paper presents a panoramic visual-inertial simultaneous localization and mapping (SLAM) system that is tightly coupled with a wheel encoder, which can be used in a mobile mapping system (MMS), robot, or driverless car. The whole SLAM system is made up of four modules: 1) measurement preprocessing; 2) system initialization; 3) tracking; and 4) local mapping. The core of the system is the combined adjustment we propose for the observations of the panoramic camera, inertial measurement unit (IMU), and wheel encoder, to optimize the system state, which includes the rotation, translation, velocity, IMU bias, and local map point coordinates in the initialization, tracking, and local mapping modules. In the preprocessing, the most important contribution is that we derive the wheel preintegration based on a two-wheel differential model, which combines the many wheel measurements between two frames into a single relative motion constraint. A novel initialization algorithm is also introduced, which can be divided into two steps: 1) the first step is to initialize the visual scale and parameters of the IMU; and 2) the second step is to perform the combined adjustment for the camera, IMU, and preintegrated wheel encoder data, to refine the initial parameters. The proposed panoramic camera-IMU-wheel SLAM (PIW-SLAM) system can achieve high-precision and robust localization in challenging scenes through multi-sensor fusion. We validated the performance of the proposed system on seven different challenging data sequences from the University of Michigan North Campus Long-Term Vision and LiDAR Dataset (NCLT) dataset. The results showed that the PIW-SLAM system has a higher localization accuracy than the other state-of-the-art SLAM systems, and it also showed a superior robustness in various complex environments. We also verified the accuracy of the scale obtained by the initialization algorithm. Furthermore, we confirmed the performance of the proposed PIW-SLAM system in a wide-baseline scene of the ground motion measurement system.

Wide-area Operation Monitoring of Conveyors Using a Panoramic Vibration Camera

In this study, we propose a novel non-contact vision sensing method for wide-area monitoring of the operation of conveyors in ironworks by using a panoramic vibration camera in real time. This method can capture magnified images, including vibrations from a single camera with mirror-driven viewpoint switching. The rotation of multiple rollers with conveyor belts was detected using a function of full-pixel vibration spectrum imaging, which can calculate peak frequencies from time frequency responses. Through experiments in different situations, such as loading and unloading, we evaluated the efficiency of this method, which can monitor the operation of multiple rollers and conveyors, when the camera is located 15 m away, or more, from the conveyors to be monitored.

Wavelet Transform-Based High-Definition Map Construction From a Panoramic Camera

Wavelet Transform-Based High-Definition Map Construction From a Panoramic Camera

Mars-rover cameras evaluation of laboratory spectra of Fe-bearing Mars analog samples

The cameras on the Pathfinder probe (Imager for Pathfinder) and the rovers Spirit and Opportunity (Panoramic Camera), Curiosity (Mast Camera) and Perseverance (Mast Camera-Z) produce visible-range spectra of limited wavelength resolution but of great target resolution which allows mineralogical analysis of selected areas within martian rocks. Laboratory spectra of relevant specimens were transformed into the spectra corresponding to each of the above cameras to increase our capability to interpret martian spectral data collected in-situ. The focus was on finding spectral features that can be detected by the cameras on Mars and are diagnostic of specific minerals. The samples are a collection of (1) Fe/Mg-phyllosilicates from submarine hydrothermal sites and (2) of rocks from acid alteration environments containing goethite, hematite, jarosite and Fe-bearing chlorite as these minerals are detectable in the extended visible range. Among all the samples, interstratified glauconite-nontronite has the most unique spectral features and should be easily detectable with the rover cameras. The spectral features of talc from Fe-bearing interstratified specimens are described. These data are especially relevant as glauconite and talc have been proposed to be fairly abundant on Mars and their detections are suggested from remote-sensing near-IR data. Several indices are proposed to assess Fe content on the investigated samples as well as mineral concentration of goethite and hematite. Among these indices, the normalized spectral slope in the range 420–600 nm increases with total Fe content for all samples, whether phyllosilicates, oxides or sulphates (R2 = 0.7–0.8). For pure phyllosilicates, the slope from 600 to 1010 nm decreases with increasing octahedral Fe (R2 = 0.75). An index for goethite produced excellent results assessing goethite concentration (R2 = 0.84). Of all cameras, Mast Camera reproduces the spectra with lowest fidelity and generates the poorest correlations between indices and tested variables. The other three cameras perform similarly.

A Semantic SLAM System for Catadioptric Panoramic Cameras in Dynamic Environments.

When a traditional visual SLAM system works in a dynamic environment, it will be disturbed by dynamic objects and perform poorly. In order to overcome the interference of dynamic objects, we propose a semantic SLAM system for catadioptric panoramic cameras in dynamic environments. A real-time instance segmentation network is used to detect potential moving targets in the panoramic image. In order to find the real dynamic targets, potential moving targets are verified according to the sphere’s epipolar constraints. Then, when extracting feature points, the dynamic objects in the panoramic image are masked. Only static feature points are used to estimate the pose of the panoramic camera, so as to improve the accuracy of pose estimation. In order to verify the performance of our system, experiments were conducted on public data sets. The experiments showed that in a highly dynamic environment, the accuracy of our system is significantly better than traditional algorithms. By calculating the RMSE of the absolute trajectory error, we found that our system performed up to 96.3% better than traditional SLAM. Our catadioptric panoramic camera semantic SLAM system has higher accuracy and robustness in complex dynamic environments.

Rock Fragments in Shallow Lunar Regolith: Constraints by the Lunar Penetrating Radar Onboard the Chang'E‐4 Mission

AbstractIt is generally believed that the top centimeters of mature lunar regolith should be dominated by fine particles that contain few rocky fragments, but surface fragments are occasionally visible in mature lunar regolith. Based on amplitudes of surface echoes received by the Lunar Penetrating Radar onboard the Yutu‐2 rover of the Chang'E‐4 mission, we extracted regions with abnormally high relative permittivity that is larger than 4, quantifying the abundance of dense fragments within the top ∼4 cm of the mature regolith along the rover path. We carried out high‐resolution numerical simulations for the propagation of electromagnetic waves, showing that the abnormally high relative permittivity is not caused by local topography but due to the existence of dense materials. Using high‐resolution images obtained by the panorama camera onboard the rover, we performed correlated observation for areas with abnormally high relative permittivity, confirming that some of these dense materials are exposed rocky fragments. However, most of the high relative permittivity areas are not correlated with surface fragments, and some are located beneath floors of highly degraded small impact craters. We found that regolith clumps formed due to impact compaction are not dense enough to cause the abnormally large relative permittivity. Alternatively, the detected dense materials may be melt‐bearing impact breccias in the shallow regolith, which were formed by the host craters. The results revealed a highly heterogeneous structure at shallow depths of mature lunar regolith, revealing a new mechanism of sand‐blasting impacts on the destruction of pores in mature lunar regolith.

Design of the Payload Subsystem of Chang’E-5 Lunar Explorer

To achieve the scientific and engineering goals for situ exploration of the sampling area, The payload subsystem of Chang'E-5 explorer is equipped with four types of scientific payloads: the panoramic camera, descending camera, lunar regolith penetrating radar, lunar mineral spectrometer. It is also equipped with the national flag of China exhibition system, the panoramic camera pan-tilt table, and payload data processor as engineering and supporting payloads. This paper introduces the scientific and engineering goals, system composition, function and performance, system design, and operating achievements of the Chang'E-5 payload subsystem.

IMAGE-BASED ORIENTATION DETERMINATION OF MOBILE SENSOR PLATFORMS

Abstract. Estimating the pose of a mobile robotic platform is a challenging task, especially when the pose needs to be estimated in a global or local reference frame and when the estimation has to be performed while the platform is moving. While the position of a platform can be measured directly via modern tachymetry or with the help of a global positioning service GNSS, the absolute platform orientation is harder to derive. Most often, only the relative orientation is estimated with the help of a sensor mounted on the robotic platform such as an IMU, with one or multiple cameras, with a laser scanner or with a combination of any of those. Then, a sensor fusion of the relative orientation and the absolute position is performed. In this work, an additional approach is presented: first, an image-based relative pose estimation with frames from a panoramic camera using a state-of-the-art visual odometry implementation is performed. Secondly, the position of the platform in a reference system is estimated using motorized tachymetry. Lastly, the absolute orientation is calculated using a visual marker, which is placed in the space, where the robotic platform is moving. The marker can be detected in the camera frame and since the position of this marker is known in the reference system, the absolute pose can be estimated. To improve the absolute pose estimation, a sensor fusion is conducted. Results with a Lego model train as a mobile platform show, that the trajectory of the absolute pose calculated independently with four different markers have a deviation < 0.66 degrees 50% of the time and that the average difference is < 1.17 degrees. The implementation is based on the popular Robotic Operating System ROS.

DEVELOPMENT OF PANORAMIC VIRTUAL TOURS SYSTEM BASED ON LOW COST DEVICES

Abstract. The virtual visit is a simulation of reality that allows virtually changing time or placing. Unlike this, virtual reality offers the possibility to associate the real world with the virtual world, while ensuring interactivity and an improvement in prospecting the reality. One of the most used form of virtual reality is the 360° panoramic imagery and video, currently in the market we can find many providers of panoramic cameras, going from costly professional cameras to the low cost consumer grade cameras. In addition, many software providers offers the virtual tours creation and visualisation, to our knowledge, only google street view offers full mapping functionalities, but the data capture and collection is an opaque operation. Generally, smartphones are equipped with positioning capabilities, based on sensors like gyroscopes, accelerometers, magnetometer used as compass and GPS, the integration of this data allow the estimation of the pose.This paper discusses the creation of an interactive virtual reality with 360° panoramic images with the possibility of automating its acquisition and integration in mapping environment by using smartphone positioning methods, while the panoramic images are taken using costumer grade camera, which composes a low cost system for acquiring and diffusing panoramic virtual tours map. The developed solution is a dynamic web interface using several open source libraries and programming tools.

A novel fusing semantic- and appearance-based descriptors for visual loop closure detection

Loop-closure detection plays an important role in visual simultaneous localisation and mapping;it is an independent part of the visual odometer and can effectively reduce its accumulated error, in addition to helping with loop-closure detection for relocalisation. With the development of deep learning methods in recent years, the training models of convolutional neural networks for major data sets have been improved for loop-closure detection. Presently, some high-level engineering problems still rely on auxiliary equipment, such as panoramic cameras and radar lasers, which greatly increase the expensive extra cost; however, owing to the extreme appearance and viewpoint changes involved in such problems, loop-closure detection that relies on two-dimensional images is not applicable. Based on the two nearest neighbour vector of locally aggregated descriptors (TNNVLAD) method, a novel feature descriptor called two nearest neighbour local sensor tensor(TNNLoST) is proposed herein by combining the semantic features of high-level neural networks with dense descriptors. This approach introduces a semantic concept similar to human cognition for the surrounding environment, thus enabling better understanding of the environment. The proposed method was applied to publicly available benchmark datasets to show its performance.

Multiscale spectral discrimination of poorly crystalline and intermixed alteration phases using aerial and ground-based ExoMars rover emulator data

A key goal of the ExoMars rover Rosalind Franklin is to analyze accessible hydrated mineral deposits using panoramic multiscale and multispectral imagery. We conducted a multiscale spectroscopic study on hydrothermally-altered basalt-hosted soils in the geothermal area of Námafjall in northern Iceland. Basaltic lavas here that have experienced first-order geochemical alteration produce a variety of cm-to-meter scale poorly crystalline alteration patterns. The resulting unconsolidated sediments provide a natural analogue material to investigate intimately mixed soils comprising multiple poorly crystalline hydrated phases. We use emulator instruments which replicate the capabilities of the ExoMars 2022 Panoramic Camera (PanCam), the Infrared Spectrometer for ExoMars (ISEM), and the CLose-UP Imager (CLUPI), alongside Raman, aerial, and X-Ray Fluorescence spectroscopic data to investigate how the detection of these mixed basalt-derived alteration phases varies as a function of spatial and spectral scale. We find soils at our study site to be comprised of unconsolidated sediments including Al-OH minerals, hydrated silica, and a variety of ferric oxides, all of which Rosalind Franklin will likely encounter along its traverse at Oxia Planum. We report on (i) the synergy and limitations between Mars rover instrument emulators as an integral part of mission preparation, (ii) how the mixed nature of these hydrothermally-altered soils affects resulting mineralogical interpretations at multiple scales, and (iii) geochemical inferences that can be made using ExoMars 2022 imaging emulators.

Spherical Image Generation from a Single Image by Considering Scene Symmetry

Spherical images taken in all directions (360 degrees by 180 degrees) allow the full surroundings of a subject to be represented, providing an immersive experience to viewers. Generating a spherical image from a single normal-field-of-view (NFOV) image is convenient and expands the usage scenarios considerably without relying on a specific panoramic camera or images taken from multiple directions; however, achieving such images remains a challenging and unresolved problem. The primary challenge is controlling the high degree of freedom involved in generating a wide area that includes all directions of the desired spherical image. We focus on scene symmetry, which is a basic property of the global structure of spherical images, such as rotational symmetry, plane symmetry, and asymmetry. We propose a method for generating a spherical image from a single NFOV image and controlling the degree of freedom of the generated regions using the scene symmetry. To estimate and control the scene symmetry using both a circular shift and flip of the latent image features, we incorporate the intensity of the symmetry as a latent variable into conditional variational autoencoders. Our experiments show that the proposed method can generate various plausible spherical images controlled from symmetric to asymmetric, and can reduce the reconstruction errors of the generated images based on the estimated symmetry.

Utilizing a Terrestrial Laser Scanner for 3D Luminance Measurement of Indoor Environments.

We aim to present a method to measure 3D luminance point clouds by applying the integrated high dynamic range (HDR) panoramic camera system of a terrestrial laser scanning (TLS) instrument for performing luminance measurements simultaneously with laser scanning. We present the luminance calibration of a laser scanner and assess the accuracy, color measurement properties, and dynamic range of luminance measurement achieved in the laboratory environment. In addition, we demonstrate the 3D luminance measuring process through a case study with a luminance-calibrated laser scanner. The presented method can be utilized directly as the luminance data source. A terrestrial laser scanner can be prepared, characterized, and calibrated to apply it to the simultaneous measurement of both geometry and luminance. We discuss the state and limitations of contemporary TLS technology for luminance measuring.

Determining urban material activities with a vehicle-based multi-sensor system

The authors describe methods of estimating the naturally occurring radioactivity present in an urban environment based on a vehicle-borne measurement system comprising an array of NaI(Tl) detectors, LiDAR systems, and panoramic cameras.

Vehicle Trajectory Extraction in Traffic Scene Based on Monocular Vision

Accurately extracting the trajectory information of multiple vehicles in urban traffic videos is of great significance in applications. It can help analyze and predict drivers’ behavior, count road flow, and restore accident scenarios. However, due to the complexity of the interaction between vehicles and the environment, it is still a challenging problem to accurately extract the trajectory of multi-target vehicles in urban traffic scenes, which often requires the assistance of a variety of different devices, which increases the practicality and difficulty of application. A method is proposed based on the combination of detection and tracking. This method only needs to use a video stream captured by a fixed panoramic camera and the coordinate information of several feature points in the scene to extract the speed and trajectory information of the vehicle in the scene. Different optimization methods are used in each link to reduce errors, and build a simulation environment for the test scenario, reconstruct and verify the extracted trajectory information in the simulation environment. Experiments have proved that the effect of the method is basically satisfactory. It has the advantages of fewer constraints, convenient deployment, and accurate restoration while ensuring a certain real-time performance, which has practical engineering significance.

Dynamic obstacle detection based on panoramic vision in the moving state of agricultural machineries

he manner in which the obstacles around the automatic navigation agricultural machinery can be detected and identified is of great importance in improving the safety and operation efficiency of agricultural machineries. This study was based on a panoramic camera to rapidly detect the dynamic obstacles around the moving agricultural machinery. It used the Lucas–Kanade optical flow algorithm to detect the moving obstacles in a panoramic image. On the basis of the actual farmland operation, background optical flow dynamic model was established to filter the background optical flow. The noise optical flow was filtered by combining K-means clustering segmentation algorithm and the variances of optical flow direction and of optical flow length within clusters. On the basis of the segmentation clusters, we used an external rectangular box to select foreground moving object. The principal optical flow direction in the segmentation clusters, and the distance between clusters determined whether the same foreground moving object was selected. Subsequently, we placed the corresponding combination processing into use, which can make the box select the complete foreground motion target. By processing 100 frames of the images, result showed that the average time consumption of the proposed method was 0.801 s. The accuracy rate of detecting dynamic agricultural machinery was 88.06%, the accuracy rate of detecting pedestrians was 81.61%, and the overall accuracy rate was 82.93%. Thus, the proposed method could meet the requirements of actual farmland operations and have a good instantaneity and detection results.

Panoramic Stereo Imaging of a Bionic Compound-Eye Based on Binocular Vision.

With the rapid development of the virtual reality industry, one of the bottlenecks is the scarcity of video resources. How to capture high-definition panoramic video with depth information and real-time stereo display has become a key technical problem to be solved. In this paper, the optical optimization design scheme of panoramic imaging based on binocular stereo vision is proposed. Combined with the real-time processing algorithm of multi detector mosaic panoramic stereo imaging image, a panoramic stereo real-time imaging system is developed. Firstly, the optical optimization design scheme of panoramic imaging based on binocular stereo vision is proposed, and the space coordinate calibration platform of ultra-high precision panoramic camera based on theodolite angle compensation function is constructed. The projection matrix of adjacent cameras is obtained by solving the imaging principle of binocular stereo vision. Then, a real-time registration algorithm of multi-detector mosaic image and Lucas-Kanade optical flow method based on image segmentation are proposed to realize stereo matching and depth information estimation of panoramic imaging, and the estimation results are analyzed effectively. Experimental results show that the stereo matching time of panoramic imaging is 30 ms, the registration accuracy is 0.1 pixel, the edge information of depth map is clearer, and it can meet the imaging requirements of different lighting conditions.