Panoramic Camera Research Articles

Gaze Control for Active Visual SLAM via Panoramic Cost Map

This paper focuses on improving the positioning accuracy of the visual simultaneous localization and mapping (VSLAM) through actively controlling the gaze of the positioning camera mounted on an autonomous mobile robot. We present a gaze control method for the VSLAM with a novel panoramic cost map. In the method, we add a panoramic camera besides the positioning camera to aid the gaze control of the positioning camera. The panoramic camera is modeled as a unit sphere model and the positioning camera is modeled as a pinhole model that is concentric with the unit sphere model. The panoramic camera captures a panoramic image for the evaluation of an environment. The effective factors, such as feature points and moving objectives, of the VSLAM are detected in the panoramic image. When the pinhole model pointing at a specific orientation, a cost value can be calculated according to the effective factors. After calculating the above-mentioned cost in all available orientations of the pinhole model, a panoramic cost map that is determined by the individual costs can be established. Next, a target orientation is selected according to the panoramic cost map and the positioning camera is rotated to the target orientation by a pan-tilt. The test results in different scenery show that the proposed method can improve the positioning accuracy of the VSLAM compared to stationary orientation.

Adaptive lens for foveal vision, imaging, and projection over large clear apertures.

We report an electrically tunable liquid crystal device that enables the generation of lenses the diameters of which may be dynamically changed from sub-millimeter to multiple millimeter sizes. These lenses can be created in different regions of interest over very large (above 50 mm) optical clear apertures. The approach is based on the activation of periodically spaced contacts on a single serpentine-shaped electrode with phase-shifted electrical signals. It enables a highly reconfigurable operation of locally created lenses with variable position, diameter, optical power (OP) and aberrations. The preliminary demonstration of the capabilities of the proposed device is presented here by creating a local lens, moving its center over an area of 25 mm x 25 mm, gradually changing its diameter from 1.3 mm to 4.55 mm as well as by tuning its OP value from zero up to, respectively, ≈ 40 and ≈3.5 diopters. Typical driving signals are at the order of 3.5 V. We think that such lenses can be used for ophthalmic or augmented reality applications as well as in microscopy, adaptive panoramic cameras with large distorted field of view, dynamic projection, etc.

Full-field deformation measurement of structural nodes based on panoramic camera and deep learning-based tracking method

Full-field deformation measurement of structures generally requires the aid of complex and expensive multi-camera measurement systems. A full-field structural deformation measurement method using a single panoramic camera and deep learning-based tracking algorithm is proposed. The contributions are as follows: (1) To address the problem that existing full-field image acquisition methods rely on multi-camera systems, a full-field image acquisition method based on a single panoramic camera is proposed, in which a distortion-free planar image covering the full-field of the structure is obtained by decomposing the projection in multiple directions based on the panoramic camera imaging model and the cubic projection method. (2) To solve the problem that the nodes of structures usually contain little texture and are difficult to track robustly with existing image processing methods, an object detection network with a modified tiny feature map layer and attention mechanism is applied to extract the region of interest (ROI) of each node automatically. (3) Finally, the ROIs of the identified nodes are clustered using a perceptual hashing method and then the node coordinates are calculated using a line segment detector (LSD) and intersection fitting. The proposed method is validated on a scaled model of a stadium, and the comparison to the deformation results of total station shows that the proposed method can calculate the displacement of all node at once, and the average error between the displacement results and those of the total station is 3.7 mm, which proves the practicality of the proposed method.

The Utility of RGB Color for Discrimination of Lunar Maturity and Composition

AbstractWe explore the extent to which red‐green‐blue (RGB) color images, such as those produced by a complementary metal oxide semiconductor (CMOS) Bayer‐filter camera, can be utilized for studying the maturity and composition of the lunar surface. RGB filters typically are quite broad, with considerable overlap among the three colors. Convolution of laboratory spectra for lunar samples to the RGB responsivities of the Chang'E‐3 rover's Panoramic Camera allowed determination of the correlations between color ratios (B/R, B/G, and G/R) and the sample maturity (Is/FeO) or composition (wt.% FeO or TiO2). In general, color ratios decrease as Is/FeO increases. When separate sample categories are considered, we find that the B/R ratio is a good predictor of Is/FeO for low‐Ti mare, low‐Fe highland, and moderate‐Fe highland soils. For high‐Ti mare soils, Is/FeO has little influence on the B/R ratio (due to the spectral effects of abundant ilmenite), and hence the ratio cannot be used to determine maturity. We also find that color ratios have no useful correlation with sample wt.% FeO or TiO2. Thus, in locations excluding the high‐Ti maria, RGB color data could be used to estimate soil maturity. We outline a concept for a multispectral imager based on a CMOS sensor with a Bayer‐like pattern of custom‐wavelength filters chosen specifically for lunar science applications.

An End-to-End Network for Upright Adjustment of Panoramic Images

Nowadays, panoramic images can be easily obtained by panoramic cameras. However, when the panoramic camera orientation is tilted, a non-upright panoramic image will be captured. Existing upright adjustment models focus on how to estimate more accurate camera orientation, and attribute image reconstruction to offline or post-processing tasks. To this end, we propose an online end-to-end network for upright adjustment. Our network is designed to reconstruct the image while finding the angle. Our network consists of three modules: orientation estimation, LUT online generation, and upright reconstruction. Direction estimation estimates the tilt angle of the panoramic image. Then, a converter block with upsampling function is designed to generate angle to LUT. This module can output corresponding online LUT for different input angles. Finally, a lightweight generative adversarial network (GAN) aims to generate upright images from shallow features. The experimental results show that in terms of angles, we have improved the accuracy of small angle errors. In terms of image reconstruction, In image reconstruction, we have achieved the first real-time online upright reconstruction of panoramic images using deep learning networks.

Design of optical system for all-weather and panoramic surveillance camera

设计了一款能实现全天候清晰成像的全景监控摄像光学系统。采用全景环带结构形式，该结构分为摄像头部单元和中继透镜单元两部分，摄像头部单元完成全视场目标搜索，中继透镜单元将头部单元所成的中间虚像进行二次成像会聚到探测器上。设计时采用多重结构优化方式，实现可见光及近红外双波段成像。该光学系统视场为360°×（40°~100°），焦距为−2.75 mm，<i>F</i>数为3.28。设计结果表明：系统的MTF（modulation transfer function）值在全视场处接近衍射极限，各个视场的弥散斑半径均小于所选CCD像元尺寸，畸变小于2 %，且日夜离焦量小于0.002 mm，该设计结果可满足全天候全景监控需求。

Automatic Registration of Panoramic Image and Point Cloud Based on the Shape of the Overall Ground Object

This paper presents a novel method for registering panoramic images and 3D point clouds using the shape of the overall ground object in the scene as registration primitives. Firstly, a semantic segmentation method is applied to the panoramic image to extract the ground object and remove the sky. Next, the cloth simulation filtering algorithm (CSF) is employed to eliminate the ground points in the 3D point cloud. The remaining 3D ground objects are then projected onto a two-dimensional plane using the imaging model of the panoramic camera to obtain the registration primitives. Finally, we adopt the whale algorithm to perform a coarse-to-fine registration, utilizing overlap degree and mutual information as the similarity measures. The proposed method is evaluated in four different scenes and compared with the other four registration methods. The results demonstrate that the proposed method is accurate and effective, with an average registration error of 11.48 pixels (image resolution is 11000x5500 pixels) compared to the EOPs of the system of 101.67 pixels.

A Railway Lidar Point Cloud Reconstruction Based on Target Detection and Trajectory Filtering

The traditional railway survey adopts a manual observation method, such as a total station measuring system. This method has high precision, but the amount of data is small, and the measurement efficiency is low. Manual measurement cannot meet the requirements of dynamic continuous high-precision holographic measurement during railway outages. Mobile laser scanning is a mobile mapping system based mainly on a laser scanner, inertial measurement unit (IMU) and panoramic camera. Mobile laser scanning has the advantages of high efficiency, high precision and automation. However, integrating inertial navigation data and mobile laser scanning data to obtain real 3D information about railways has always been an urgent problem to be solved. Therefore, a point cloud reconstruction method is proposed based on trajectory filtering for a mobile laser scanning system. This paper corrects the odometer data by identifying railway feature points through deep learning and uses Rauch–Tung–Striebel (RTS) filtering to optimize the trajectory results. Combined with the railway experimental track data, the maximum difference in the east and north coordinate direction can be controlled within 7 cm, and the average elevation error is 2.39 cm. This paper applies a multi-sensor integrated mobile detection system to railway detection. It is of great significance to the healthy development of the intelligent railway system.

Remote AFIS: development and validation of low-cost remote tower concepts for uncontrolled aerodromes

Remote tower systems are widely established as a means to provide efficient air traffic control (ATC) from a remote location. However, even these cost-effective systems cause procurement, implementation, and maintenance costs, which make them unaffordable for non-ATC aerodromes with low revenues, often only offering an aerodrome flight information service (AFIS) or UNICOM information service. In this paper, two more inexpensive concepts enabling remote control at these aerodromes are presented. They are based on a simplified camera set-up comprising a pan-tilt-zoom-camera (PTZ-camera) and a simple panoramic camera. A virtual reality-headset (VR-headset) is used to display the video streams and to control the PTZ-camera. The results of a validation study with nine ATC and AFIS officers using live traffic at the Braunschweig-Wolfsburg aerodrome (BWE) are presented. They are discussed with respect to perceived usability, virtual reality induced cybersickness, and operator acceptance. The system’s cost is compared to that of a conventional remote tower set-up. Furthermore, measured objective data in the form of angular head rotation velocities are presented and requirements for the camera set-up are deduced. In conclusion, the developed concept utilizing both the panorama camera and PTZ-camera was found to be sufficiently usable and accepted by the validation participants. In contrast, the concept based only on the PTZ-camera suffered from jerky and delayed camera movements leading to considerable cybersickness and making it barely usable.

Spectrophotometry of the lunar regolith using the Chang’E-3 Panoramic Camera (PCAM)

Aims. We present a multiband photometric study of the lunar mare surface based on in situ measurements using the Panoramic Camera on board the Chang'E-3 mission. We aim to obtain photometric properties and to investigate the differences in measuring sites in various conditions. Methods. Data were collected with a vast range of phase angles from 0.5° to 160°, probing 0–5° phases to constrain the opposition effect (OE) and phases above 120° to constrain the phase function. Stray light and spectral corrections were conducted to calibrate the regolith’s bidirectional reflectance distribution function. We employed the Hapke photometric model to describe the regolith’s photometric properties. Results. Phase functions and Hapke parameters at three measuring sites were retrieved. The mare regolith at the measuring sites had a single-scattering albedo (SSA) of 0.17-0.21 for the green channel and exhibited backscattering and a strong OE. The coherent backscatter opposition effect (CBOE) represented the majority of OE, with a proportion of around 80% for pristine regolith. The phase ratio curves were arch-shaped, and the arch positions of the curves were associated with grain size.

Population of Degrading Small Impact Craters in the Chang’E-4 Landing Area Using Descent and Ground Images

The landing camera (LCAM) of Chang’e-4 lander provides a series of low (46 cm/pixel) to high (2.3 cm/pixel) resolution images, which are suitable for centimeter-scale craters. In this paper, we analyze the degradation of those small-sized craters to provide detailed information on the local geological evolution of the lunar surface. From the mosaicked descent image, 6316 craters were extracted and classified into four degradation levels based on their morphology on the image: fresh, slightly degraded, moderately degraded, and severely degraded. The ground terrain camera (TCAM) image and the DEM of the Yutu-2 panoramic camera (PCAM) validate the crater degradation levels from a qualitative and quantitative perspective, respectively. The results show that the smaller the size of the craters, the more easily they are degraded. The crater populations in equilibrium in the four study areas indicate that the cumulative size–frequency distribution (SFD) slope is different from previous research results, and the smaller the craters, the more difficult to reach an equilibrium state (for craters smaller than a given size, the production rate is exactly balanced by the removal rate), which may be due to secondary cratering and surface resurfacing caused by the burial of ejecta from neighboring craters.

Lighting measurement with a 360° panoramic camera: Part 1 – Technical procedure and validation

This study developed a 360° field of view (FOV) lighting measurement method with the aid of the Ricoh Theta Z1 panoramic camera. As versatile lighting information from all viewing directions within 360° FOV can be retrieved from a single 360° high dynamic range (HDR) image, this new method improves the lighting measurement efficiency. Part 1 of this study reported in the present paper focuses on technical procedure and validation. Firstly, all technical issues were solved for using the Theta Z1 camera to conduct 360° FOV lighting measurement with a provided and validated step-by-step procedure. A new illuminance measurement method was then developed with the aid of the Theta Z1 camera for calculating all directional illuminance data at the camera measurement point from any viewing direction within the 360° panoramic FOV, retrieved from a single 360° HDR image taken in the field. This 360° FOV lighting measurement method has average error rate of 4.0% ± 2.4% for luminance measurement and 3.1% ± 2.6% for illuminance measurement in building interiors. Reported outcomes include the detailed procedure and algorithms for the configuration, calibrations and post-image processing, and corresponding MATLAB code and Python programs are shared online.

Registration of Terrestrial LiDAR and Panoramic Imagery Using the Spherical Epipolar Line and Spherical Absolute Orientation Model

Registration between terrestrial LiDAR and optical imagery plays a crucial role in information fusion. However, it is difficult to find reliable correlations among the different feature information of optical imagery and LiDAR point clouds. Therefore, in order to achieve high-precision registration of heterogeneous sensors, a method based on spherical epipolar line and spherical absolute orientation is proposed in this paper. The method firstly projects the LiDAR point clouds into spherical images based on the spherical imaging model and derives the spherical epipolar line equation. Then the relative and absolute orientations of the spherical LiDAR images and the optical images are performed based on manually selected control points. Finally, based on Harris corner extraction, combined with the geometric constraints of the spherical epipolar line and absolute orientation, dense matching between optical and LIDAR images are achieved, and all matching points are used as control points for registration to improve the accuracy of manually selected points registration. Multiple sets of test data are acquired outdoors using a FARO Focus S laser scanner, a Z + F IMAGER 5010C laser scanner, and a Ladybug5+ panoramic camera. The experimental results show that the method in this paper is practical and improves the accuracy of manual points selection registration, and the degree of improvement is related to the number of successfully matched corner points.

Lighting measurement with a 360° panoramic camera: Part 2 – Applications

This study developed a 360° field of view (FOV) lighting measurement method with the aid of the Ricoh Theta Z1 panoramic camera. As versatile lighting information from all viewing directions within 360° FOV can be retrieved from a single 360° high dynamic range image, this new method improves the lighting measurement efficiency. Potential unique applications are reported in the present paper as Part 2 of this study. Corresponding lighting information covers a 360° luminance map, a 360° coefficient of variation map of luminance (CVL), a 360° luminance contrast ( C1) map or ratio ( Cr1) map between any interested target located at the viewing direction and its near background, a 360° luminance contrast ( C2) map or ratio ( Cr2) map between the same target and the ambient environment and a 360° illuminance map and its single coefficient of variation of illuminance (CVE) value of all illuminances on the illuminance map. All lighting information is measured at the same point with 360° viewing directions to analyse the uniformity and continuity of the lighting environment. An angular distance weighted illuminance calibration is used to correct possible mismatched white balance of the front and rear camera lenses in a scenario with multiple light sources of dramatically different Correlated Colour Temperature (CCTs). All customized camera-control and data treatment programs, including MATLAB code and Python programs, are uploaded to the project GitHub site for online sharing.

Optimizing ExoMars Rover Remote Sensing Multispectral Science: Cross‐Rover Comparison Using Laboratory and Orbital Data

AbstractMultispectral imaging instruments have been core payload components of Mars lander and rover missions for several decades. In order to place into context the future performance of the ExoMars Rosalind Franklin rover, we have carried out a detailed analysis of the spectral performance of three visible and near‐infrared (VNIR) multispectral instruments. We have determined the root mean square error (RMSE) between the expected multispectral sampling of the instruments and high‐resolution spectral reflectance data, using both laboratory spectral libraries and Mars orbital hyperspectral data. ExoMars Panoramic Camera (PanCam) and Mars2020 Perseverance Mastcam‐Z instruments have similar values of RMSE, and are consistently lower than for Mars Science Laboratory Mastcam, across both laboratory and orbital remote sensing data sets. The performance across mineral groups is similar across all instruments, with the lowest RMSE values for hematite, basalt, and basaltic soil. Minerals with broader, or absent, absorption features in these visible wavelengths, such as olivine, saponite, and vermiculite have overall larger RMSE values. Instrument RMSE as a function of filter wavelength and bandwidth suggests that spectral parameters that use shorter wavelengths are likely to perform better. Our simulations of the spectral performance of the PanCam instrument will allow the future use of targeted filter selection during ExoMars 2022 Rosalind Franklin operations on Mars.

Rock fracturing observation based on microseismic monitoring and borehole imaging: In situ investigation in a large underground cavern under high geostress

During the excavation of deep rock engineering projects under high geostress, the brittle failure of hard rock often occurs. To understand the excavation-induced fracturing process of the surrounding rock of a large underground powerhouse, in situ observation schemes have been proposed, including microseismic (MS) monitoring, imaging with a digital panoramic borehole camera, multipoint extensometer measurements and on-site surrounding rock failure investigation. First, a MS system is used to capture the microfractures inside the rock mass, revealing the potential high-risk failure regions. The spatial distribution of MS events shows that the excavation-induced MS events are progressively clustered in the upstream spandrel and downstream foot of the powerhouse, approximately parallel to the direction of the maximum principal stress. Second, the digital panoramic borehole camera and multipoint extensometer are arranged in the high-risk region to further reflect the rock mass fracture evolution and deformation response characteristics in the monitored section. The borehole camera observations of the top arch and sidewall indicate that the excavation unloading not only causes surrounding rock fracturing near the periphery of the cavern but also induces deep fracturing at a certain depth from the periphery of the cavern. From these comprehensive observation results, the mechanism of rock mass mechanical behaviour is clarified; that is, the macroscopic deformation or failure characteristics of the surrounding rock are directly related to the evolution of microfractures. The research results of this paper are valuable for the stability analysis and excavation optimization of similar underground caverns.

ЗРИТЕЛЬ В ПРОСТРАНСТВЕ ВИРТУАЛЬНОЙ РЕАЛЬНОСТИ: ФОРМИРОВАНИЕ ПЛАНЕТАРНОЙ ОПТИКИ

ЗРИТЕЛЬ В ПРОСТРАНСТВЕ ВИРТУАЛЬНОЙ РЕАЛЬНОСТИ: ФОРМИРОВАНИЕ ПЛАНЕТАРНОЙ ОПТИКИ

Egocentric Two-frame Pedestrian Trajectory Prediction Algorithm Based on a Panoramic Camera

With the development of service industry, service-oriented mobile robot navigation technology has received more and more attention. In order to improve the obstacle avoidance performance of service-oriented mobile robots, accurate pedestrian trajectory prediction is essential. Therefore, we propose an egocentric two-frame pedestrian trajectory prediction algorithm based on a panoramic camera, the Future Social Prediction (FSP) algorithm, to solve the pedestrian trajectory prediction problem in the human-robot coexistence environment. The input of FSP is the information in the two frames of a panoramic camera. First, the free pedestrian prediction network (FPPN) is used to predict the free movement of pedestrians. Then the future social pooling module (FSPM) is used to process the prediction results of FPPN. And finally, the social pedestrian prediction network (SPPN) combines the frame information with the output of FSPM to get the final prediction results. A panoramic camera is used as a sensor to reduce blind spots in the field of view. Only two frames are used for prediction to improve the real-time performance of FSP. We record the panoramic video and create a dataset. FSP is tested on our dataset with other two algorithms, and the results show that FSP has better performance than Spatio-Temporal Encoder-Decoder model (STED) and Future Person Localization (FPL).

Mechanical Particle Filter-Based Active Vision System for Fast Wide-Area Multiobject Detection

Sliding window searching is a common paradigm employed for large Field-of-View (FOV) multi-object detection. However, low target resolution and irregular distribution of objects make the traditional method inefficient and hard to meet the demands of fast object detection in a wide area. To address the above issues, in this paper we design an ultrafast switching galvano-mirror based active vision system that can simultaneously search and detect multi-object at 300 frames per second (fps) in a large FOV, without the guidance of a high-resolution (HR) panoramic camera. Specifically, a novel mechanical particle filter (MPF) framework is proposed to improve the efficiency of searching potential objects by constructing the probability distribution model and then iteratively locating target objects, where each particle is represented by an image of a single small FOV that is controlled and captured by a switching galvano-mirror. The object detector is used to determine if the particle image contains target objects. Then, to reduce the total scanning cost for capturing all particles, we introduce a center partitioned scanning algorithm that significantly speeds up the scanning process by 10&#x00D7; or more. Moreover, we optimize the processing pipeline for low latency sensing by using a high-parallelism visual feedback scheme to improve efficiency. Finally, an efficient and fast active vision system is implemented, which is based on a 2-axis galvano-mirror and a high-speed camera without multi-camera fusion for active object searching. Abundant experimental results demonstrate the effectiveness of our proposed system in indoor environments and real-world scenarios. We also verify the performance in terms of scanning speed and detection accuracy.

Defect Detection for a Vertical Shaft Surface Based on Multimodal Sensors

Hydroelectricity is a major source of renewable electricity originating from a turbine driven by dammed large-volume water via a penstock or a drop shaft. Shafts suffer from risks of collapse due to the pressure from exterior structures and the erosion from inner water flow. Vertical shafts are an important part of hydroelectric power generation systems, and detecting defects in shafts guarantees the stable operation of hydropower stations. However, shaft defect detection is a great challenge due to the poor conditions, large drop in height, limited entrance size, lack of light and damp air, where suitable technology and equipment are not available. Aiming at defect detection for vertical shafts, we have developed a defect-detecting system based on unmanned airships, integrated panoramic CCD cameras, 3D laser scanners, inertial measurement units, barometric altimeters, illumination sensors, and control modules. Shaft defect detection methods (SDDMs) are proposed by fusing the multi-modal image features to extract typical defects on concrete surfaces. Compared with machine learning methods, the proposed method achieves the highest overall accuracy of 90.90&#x0025; for defect detection. Our system was validated by experiments in the shafts of the Nuozhadu hydropower station to be functional for defect detection, which demonstrates its capability of reducing the risk of collapse and improving safety.