Perspective Projection Research Articles

EPMF: Efficient Perception-Aware Multi-Sensor Fusion for 3D Semantic Segmentation.

We study multi-sensor fusion for 3D semantic segmentation that is important to scene understanding for many applications, such as autonomous driving and robotics. Existing fusion-based methods, however, may not achieve promising performance due to the vast difference between the two modalities. In this work, we investigate a collaborative fusion scheme called perception-aware multi-sensor fusion (PMF) to effectively exploit perceptual information from two modalities, namely, appearance information from RGB images and spatio-depth information from point clouds. To this end, we project point clouds to the camera coordinate using perspective projection, and process both inputs from LiDAR and cameras in 2D space while preventing the information loss of RGB images. Then, we propose a two-stream network to extract features from the two modalities, separately. The extracted features are fused by effective residual-based fusion modules. Moreover, we introduce additional perception-aware losses to measure the perceptual difference between the two modalities. Last, we propose an improved version of PMF, i.e., EPMF, which is more efficient and effective by optimizing data pre-processing and network architecture under perspective projection. Specifically, we propose cross-modal alignment and cropping to obtain tight inputs and reduce unnecessary computational costs. We then explore more efficient contextual modules under perspective projection and fuse the LiDAR features into the camera stream to boost the performance of the two-stream network. Extensive experiments on benchmark data sets show the superiority of our method. For example, on nuScenes test set, our EPMF outperforms the state-of-the-art method, i.e., RangeFormer, by 0.9% in mIoU.

Blind 3D Video Stabilization with Spatio-Temporally Varying Motion Blur

Video stabilization is a challenging task that attempts to compensate for the overall frame shake during video acquisition. Existing three-dimensional video stabilization methods aim at modeling camera perspective projection through either data-driven training or explicit motion estimation. However, the above methods are difficult to effectively solve the issue of shaky videos with abrupt object movements, resulting in local motion blur in the direction of the movement. This phenomenon is prevalent in real-world scenarios featuring foreground blind motion scenes. Unfortunately, directly combining stabilization and deblurring methods poses challenges when dealing with this situation. In the video, the intensity of motion blur undergoes continuous changes, and the direct combination method inadequately utilizes spatiotemporal information, providing insufficient clues for cross-frame compensation. To alleviate this problem, the Cross-frame-temporal Module framework is proposed to address blind motion blur induced by various conditions, which utilizes cross-frame temporal features to estimate depth maps and camera motion. In this framework, a Blur Transform Network (BTNet) is designed to adapt to spatially varying motion blur, which transforms local regions according to the impact of blur intensities to adapt to the effects of non-uniform motion blur; furthermore, our Temporal-Aware Network (TANet) further suppresses motion blur by leveraging cross-frame temporal features. In addition, the limited availability of pair-training video data containing motion blur limits the application of this approach in practice. The Cross-frame-temporal Module framework adopts an un-pretrained in-test training strategy. Extensive experimental results have demonstrated that our method outperforms state-of-the-art methods.

Drift compensation system of an autocollimator based on reciprocal roll angles and a feedback combination target

To address the issues of limited field of view and beam drift in traditional autocollimation measurements, a novel, to our knowledge, autocollimation measurement method is proposed. This method is based on the principle of full-circle continuous measurement using reciprocal roll angles of a compound biaxial system, combined with a feedback and compensation mechanism for beam drift using both calibration and compensation targets. A self-collimating small-angle measurement system model, coupled with a compound biaxial turntable and combined target, is established using ray tracing and perspective projection transformation techniques. Additionally, a closed-loop beam drift compensation system is developed by applying a BP neural network and a PID control algorithm. Experimental tests on the compensation system demonstrate a significant improvement in measurement accuracy and stability. The results show that this method satisfies the requirements for small-angle autocollimation measurement, and the method also enhances the overall stability of the autocollimation system.

Adaptive positioning of large-scale aircraft components using a simplified image-based visual servoing via online measurement of ball-head center

In digital alignment systems, ensuring fast and accurate alignment between the ball head on large aircraft components and the ball socket of the positioner is the key to aircraft assembly efficiency and quality. Image-based visual servoing is one of the most advanced techniques with high efficiency and positioning accuracy, but it does not account for measuring the ball-head position. Moreover, variable target positions, similar feature interference, and poor lighting conditions pose great challenges to the fast and robust visual measurement of ball-head position. Hence, an online visual measurement method is introduced to measure the absolute position of the ball head in real time. An adaptive region of interest extraction algorithm and an improved edge following method are introduced to extract the elliptical contour from the ball-head image, and ellipse detection is achieved through a simple triplet selection algorithm. A geometric model based on the perspective projection of the sphere is established for estimating the ball-head center. Then, the simplified image-based visual servoing method is developed by constantizing the image Jacobian matrix. Next, an adaptive positioning method of large-scale aircraft components with multiple ball heads is proposed from a holistic perspective. Finally, experiments show that the final positioning accuracy of each ball head in the X/Y plane is less than ± 0.05 mm, with an efficiency almost ten times that of the conventional method. After adaptive adjustment, the position and angle errors of the simulated component are reduced within 0.6 mm and 0.3°, respectively.

Perspective Transform-based Depth Estimation of Monocular Camera for Electrocution Threat Determination of Construction Machinery

Abstract. This paper discusses the application of perspective transform-based monocular camera depth estimation method in monitoring the safety distance between construction machinery and power lines. The traditional distance measurement relies on manual operation, which is inefficient and not precise enough, while the monocular depth estimation method based on deep learning can improve the accuracy, but faces high equipment cost and relies on a large amount of training data. In contrast, the depth estimation method based on perspective transformation proposed in this paper utilizes mathematical and physical principles to establish a mathematical model between the pixel distances of an image and the actual scene distances through perspective projection theory, which simplifies the hardware requirements and reduces the data dependence and improves the computational efficiency and applicability. Through experimental validation under different construction scenes and exposure conditions, the method demonstrates high accuracy and robustness, proving its practicality in construction safety management. This research provides a new technical direction and practical application possibility in the field of intelligent monitoring and 3D reconstruction.

Plastic Deposition in Sediments of Moreton Bay, Australia: A Historical Perspective and Potential Future Projections

Plastic Deposition in Sediments of Moreton Bay, Australia: A Historical Perspective and Potential Future Projections

A review of general perspective projections onto a cylinder

A review of general perspective projections onto a cylinder

Regiomontanus angle maximization for catadioptric sensors with paraboloidal mirrors

Dioptric cameras with conventional perspective projection have well established analytical properties. However, they suffer from perspective distortions and only have a limited field of view. Catadioptric cameras offer panoramic imaging. Their extensive field of view together with projection specific image analysis, can simplify many computer vision tasks. Several properties of catadioptric projection for geometric primitives such as points and lines have been addressed and have also been used for calibration. However, higher order geometric properties are yet to be investigated. Such analysis is complicated by the specifics of the warping of the scene by catadioptric projection. One such property, that is the subject of this work, is the Regiomontanus angle maximization relative to the effective viewpoint of the sensor. This work considers catadioptric sensors with paraboloidal mirrors, that is, paracatadioptric sensors. Analytical ray tracing of a simplified 1D world object gives its projection in the image and an expression for its length. The optimization of the length of the projection results in a third degree equation for the Regiomontanus distance that can be solved explicitly. The Khayyam geometric solution of this equation provides the Regiomontanus distance of maximum subtended projection for these cameras. Applications of these results in various contexts are presented and discussed.

Climbing robot-assisted quantitative identification of sheath surface defects of stay cables

Sheath surface defect identification of stay cables using climbing robots or drones is gaining popularity. However, effective quantitative identification approaches remain challenging. This study proposes a cable climbing robot assisted quantitative identification method to address sheath surface defects combining the deep learning method and image processing techniques. First, standard and high-resolution cable sheath images, collected by an improved cable climbing robot with exclusive scales indicating the defect size, were categorised into three datasets according to defect types. Subsequently, a mask region-based conventional neural network (Mask R-CNN) model was developed for defect capture, with a connected domain algorithm quantifying defect areas. Perspective projection was applied to the defect masks to reduce the image distortion effect. The test results show that the model can achieve over 89% precision, recall and F 1-score for three defect types, with masks presenting overlapping rates reach up to 87% compared to the ground-truth regions. The quantitative identification results are promising to facilitate stay cable assessment and maintenance. Future study will be focused on extending the volume and categories of the dataset and improving the performance of the model.

Polarimetric Helmholtz Stereopsis.

Helmholtz stereopsis (HS) exploits the reciprocity principle of light propagation (i.e., the Helmholtz reciprocity) for 3D reconstruction of surfaces with arbitrary reflectance. In this paper, we present the polarimetric Helmholtz stereopsis (polar-HS), which extends the classical HS by considering the polarization state of light in the reciprocal paths. With the additional phase information from polarization, polar-HS requires only one reciprocal image pair. We derive the reciprocity relationship of Mueller matrix and formulate new reciprocity constraint that takes polarization state into account. We also utilize polarimetric constraints and extend them to the case of perspective projection. For the recovery of surface depths and normals, we incorporate reciprocity constraint with diffuse/specular polarimetric constraints in a unified optimization framework. For depth estimation, we further propose to utilize the consistency of diffuse angle of polarization. For normal estimation, we develop a normal refinement strategy based on degree of linear polarization. Using a hardware prototype, we show that our approach produces high-quality 3D reconstruction for different types of surfaces, ranging from diffuse to highly specular.

High-precision polarization imaging for three-dimensional reconstruction aided by a separate coarse depth map

For less-texture objects with highly reflective regions, traditional vision-based three-dimensional reconstruction techniques often fail to yield ideal results. Utilizing polarization information for reconstructing such objects is a convenient and effective method. However, relying solely on polarization information for three-dimensional reconstruction presents challenges such as ambiguity in surface normals and difficulties in normal integration. In this paper, we propose to resolve the ambiguity in polarization normals using the coarse depth as prior information under a perspective projection model. By fusing the disambiguated normal with the coarse depth, we avoid the need for normal integration. The experimental results demonstrate that our proposed algorithm improves the quality of polarization imaging, effectively restoring the details lost in the coarse depth and smoothing the areas with high reflectivity.

Correction for geometric distortion in the flattened representation of pipeline external surface

Conventional cameras often produce external pipeline surface images with significant perspective-projection distortions, making determining the type and size of defects on the pipeline surface difficult. To address this issue, we present a novel approach for correcting the geometric distortion of a constant diameter pipeline (CDP) external surface image based on apparent contour pairs (ACPs). First, the camera is calibrated and the real projection position of ACPs accurately extracted. Second, a single-view perspective projection model of the CDP is adopted to realize a three-dimensional reconstruction of the pipeline. Subsequently, the flattened image of the CDP’s outer surface is obtained by performing equidistant sampling of cross sections, determining the visibility of discrete points, and mapping projection points onto a flat surface. Simulations and real experiments demonstrated the high accurate method in the correction of CDP images. Experiments conducted under different outdoor scenarios confirmed the robustness of the proposed method.

Place and Independence Are Formed by Moving Furniture

In contemporary architectural design, a noticeable trend emerges in the meticulous planning of common areas, with an emphasis on fostering the free movements of users within these areas. It is important to consider the architectural elements that influence user behavior in these environments. Previous studies have delved into the relationship between architecture and user activity, often through questionnaires and activity surveys conducted in arbitrary spaces. However, such data are discontinuous and only capture a portion of the space. Therefore, in this study, continuous and extensive recordings were performed in an arbitrary space using a video camera to capture the entirety of the area. Consequently, we were able to comprehend the relationship between architecture and people’s activities and observe the formation of user autonomy through their decisions to rearrange furniture. Furthermore, an “Open Pose” system was used to detect people and generate a user stay distribution chart. This was achieved by converting it into a plan view using perspective projection transformation, thereby obtaining objective data automatically. Consequently, we verified the feasibility of conducting visually comprehensible processing using tools like “Open Pose”, which was used in one segment of the experiment.

Improving Measurement Accuracy of Deep Hole Measurement Instruments through Perspective Transformation.

Deep hole measurement is a crucial step in both deep hole machining and deep hole maintenance. Single-camera vision presents promising prospects in deep hole measurement due to its simple structure and low-cost advantages. However, the measurement error caused by the heating of the imaging sensor makes it difficult to achieve the ideal measurement accuracy. To compensate for measurement errors induced by imaging sensor heating, this study proposes an error compensation method for laser and vision-based deep hole measurement instruments. This method predicts the pixel displacement of the entire field of view using the pixel displacement of fixed targets within the camera's field of view and compensates for measurement errors through a perspective transformation. Theoretical analysis indicates that the perspective projection matrix changes due to the heating of the imaging sensor, which causes the thermally induced measurement error of the camera. By analyzing the displacement of the fixed target point, it is possible to monitor changes in the perspective projection matrix and thus compensate for camera measurement errors. In compensation experiments, using target displacement effectively predicts pixel drift in the pixel coordinate system. After compensation, the pixel error was suppressed from 1.99 pixels to 0.393 pixels. Repetitive measurement tests of the deep hole measurement instrument validate the practicality and reliability of compensating for thermal-induced errors using perspective transformation.

Polar-vision1: A Novel Collinearity Equation of Perspective Projection in Polar Coordinate System

Abstract. Progress has been made in the community of photogrammetry and 3d computer vision in addressing the mathematical challenge posed by the collinearity equation. We introduce a new method for establishing the coordinate reference for 2d pixels and 3d landmarks using 'angular coordinates'. The mathematical relationships required for converting 3d landmarks, expressed in angular coordinates, to the camera framework are presented. The landmarks are then projected using perspective projection to obtain 2d pixels represented in angular coordinates. This framework is formally nominated as the 'Polar-vision1', which has been developed and integrated into the commercial software G3D-Cluster. Its application to pinhole camera image processing has demonstrated superior efficiency and admission rates of tie points, as well as reconstruction detail capabilities, compared to OpenMVG, achieving approximately a 1.4× improvement. The project 'Key Technologies and Tool System for Realistic 3D Modeling through Integration of Multi-Source Information in the Space-Air-Ground Domain' was awarded First Prize at the 2023 Surveying Science and Technology Awards, with Polar-vision1 as one of the innovative points.

IModel: an interactive 3D crystal structure visualization program

This work presents present iModel, an interactive 3D crystal structure visualization program capable of reading crystal structure datafiles in various formats and rendering corresponding 3D models. Users can interact with the models by translating, rotating and zooming in on them. The software visualizes crystal structures, including atoms, bonds, unit cells and polyhedral shapes using spheres, lines and polyhedra, with customizable colors and sizes for individual atoms and bonds. The software features perspective and orthogonal projections, multiple chemical bond search modes, and a user-friendly toolbar for view control, and exports models as images or saves progress in project files. Developed on the LabVIEW platform with Python extensions, the iModel program leverages a queue message handler and producer–consumer pattern for a modular, reusable and powerful crystal structure visualization tool.

Toward a theory of perspective perception in pictures.

This paper reviews projection models and their perception in realistic pictures, and proposes hypotheses for three-dimensional (3D) shape and space perception in pictures. In these hypotheses, eye fixations, and foveal vision play a central role. Many past theories and experimental studies focus solely on linear perspective. Yet, these theories fail to explain many important perceptual phenomena, including the effectiveness of nonlinear projections. Indeed, few classical paintings strictly obey linear perspective, nor do the best distortion-avoidance techniques for wide-angle computational photography. The hypotheses here employ a two-stage model for 3D human vision. When viewing a picture, the first stage perceives 3D shape for the current gaze. Each fixation has its own perspective projection, but, owing to the nature of foveal and peripheral vision, shape information is obtained primarily for a small region of the picture around the fixation. As a viewer moves their eyes, the second stage continually integrates some of the per-gaze information into an overall interpretation of a picture. The interpretation need not be geometrically stable or consistent over time. It is argued that this framework could explain many disparate pictorial phenomena, including different projection styles throughout art history and computational photography, while being consistent with the constraints of human 3D vision. The paper reviews open questions and suggests new studies to explore these hypotheses.

Metric localization for lunar rovers via cross-view image matching

Accurate localization is critical for lunar rovers exploring lunar terrain features. Traditionally, lunar rover localization relies on sensor data from odometers, inertial measurement units and stereo cameras. However, localization errors accumulate over long traverses, limiting the rover’s localization accuracy. This paper presents a metric localization framework based on cross-view images (ground view from a rover and air view from an orbiter) to eliminate accumulated localization errors. First, we employ perspective projection to reduce the geometric differences in cross-view images. Then, we propose an image-based metric localization network to extract image features and generate a location heatmap. This heatmap serves as the basis for accurate estimation of query locations. We also create the first large-area lunar cross-view image (Lunar-CV) dataset to evaluate the localization performance. This dataset consists of 30 digital orthophoto maps (DOMs) with a resolution of 7 m/pix, collected by the Chang’e-2 lunar orbiter, along with 8100 simulated rover panoramas. Experimental results on the Lunar-CV dataset demonstrate the superior performance of our proposed framework. Compared to the second best method, our method significantly reduces the average localization error by 26% and the median localization error by 22%.

Soft Tactile Sensing for Object Classification and Fine Grasping Adjustment Using a Pneumatic Hand With an Inflatable Palm

Power grasps and fine manipulations with a robotic hand usually require dexterity in the fingers and a rich tactile function. In this paper, we propose a novel soft hand with a tactile sensing function and investigate how tactile information can be used to perform object classification and fine grasping adjustments. We first developed a pneumatically driven soft hand with an inflatable palm and bendable fingers, where a TacTip -type sensor and several flex sensors were integrated. The inflatable palm can not only significantly improve grasping but also object sensing. Based on the perspective projection model of a camera, a 3D reconstruction algorithm for the deformed surface of a soft palm is presented. A spatial characterization of geometric information regarding objects based on sensing information from the palm and fingers was deduced and analyzed. A classifier was designed for object classification. Finally, a control method for the fine adjustment of grasps based on tactile sensing information was presented and verified using a series of grasps of different objects with a soft hand. The object classification and contact state sensed by the soft tactile sensors can be used for grasp planning and manipulation control with the soft hand.

The Karelian Arctic in the directions of China’s Arctic policy at the present stage

The article identifies four main areas that define China’s strategic goals in the Arctic region: the energy sector, infrastructure, tourism, educational and scientific co-operation. Special attention is paid to the strategic friendship between Russia and China in the context of Arctic policy. The RussianChinese partnership in this region is seen as a key factor in ensuring the stability and development of Russia’s Arctic zone. The interaction between the two countries not only contributes to the strengthening of economic ties, but also ensures the security and sustainable development of the region. The paper presents examples of successful cooperation between the Republic of Karelia and the People’s Republic of China and analyses these relations as an example of the practical implementation of China’s Arctic strategy at the regional level. It is important to note that the realisation of many investment projects on exploration and mining in the Karelian Arctic is connected with the attraction of Chinese investment in this industry. The article also evaluates Chinese interests in the Karelian Arctic in other areas, such as the construction of hydroelectric power station infrastructure projects in the Kemsky region as well as some perspective projects related to the construction of a deep-sea port in Kem or a seaport in Belomorsk.