Multi-view Scene Research Articles

Multi-view scene matching with relation aware feature perception

For scene matching, the extraction of metric features is a challenging task in the face of multi-source and multi-view scenes. Aiming at the requirements of multi-source and multi-view scene matching, a siamese network model for Spatial Relation Aware feature perception and fusion is proposed. The key contributions of this work are as follows: (1) Seeking to enhance the coherence of multi-view image features, we investigate the relation aware feature perception. With the help of spatial relation vector decomposition, the distribution consistency perception of image features in the horizontal H→ and vertical W→ directions is realized. (2) In order to establish the metric consistency relationship, the large-scale local information perception strategy is studied to realize the relative trade-off scale selection under the size of mainstream aerial images and satellite images. (3) After obtaining the multi-scale metric features, in order to improve the metric confidence, the feature selection and fusion strategy is proposed. The significance of distinct feature levels in the backbone network is systematically assessed prior to fusion, leading to an enhancement in the representation of pivotal components within the metric features during the fusion process. The experimental results obtained from the University-1652 dataset and the collected real scene data affirm the efficacy of the proposed method in enhancing the reliability of the metric model. The demonstrated effectiveness of this method suggests its applicability to diverse scene matching tasks.

A Contrastive Learning Based Multiview Scene Matching Method for UAV View Geo-Localization

Multi-view scene matching refers to the establishment of a mapping relationship between images captured from different perspectives, such as those taken by unmanned aerial vehicles (UAVs) and satellites. This technology is crucial for the geo-localization of UAV views. However, the geometric discrepancies between images from different perspectives, combined with the inherent computational constraints of UAVs, present significant challenges for matching UAV and satellite images. Additionally, the imbalance of positive and negative samples between drone and satellite images during model training can lead to instability. To address these challenges, this study proposes a novel and efficient cross-view geo-localization framework called MSM-Transformer. The framework employs the Dual Attention Vision Transformer (DaViT) as the core architecture for feature extraction, which significantly enhances the modeling capacity for global features and the contextual relevance of adjacent regions. The weight-sharing mechanism in MSM-Transformer effectively reduces model complexity, making it highly suitable for deployment on embedded devices such as UAVs and satellites. Furthermore, the framework introduces a contrastive learning-based Symmetric Decoupled Contrastive Learning (DCL) loss function, which effectively mitigates the issue of sample imbalance between satellite and UAV images. Experimental validation on the University-1652 dataset demonstrates that MSM-Transformer achieves outstanding performance, delivering optimal matching results with a minimal number of parameters.

Radar Feature Analysis of Human Activity Recognition Under Multiview Scenes

Radar Feature Analysis of Human Activity Recognition Under Multiview Scenes

Multi-View Scene Classification Based on Feature Integration and Evidence Decision Fusion

Leveraging multi-view remote sensing images in scene classification tasks significantly enhances the precision of such classifications. This approach, however, poses challenges due to the simultaneous use of multi-view images, which often leads to a misalignment between the visual content and semantic labels, thus complicating the classification process. In addition, as the number of image viewpoints increases, the quality problem for remote sensing images further limits the effectiveness of multi-view image classification. Traditional scene classification methods predominantly employ SoftMax deep learning techniques, which lack the capability to assess the quality of remote sensing images or to provide explicit explanations for the network’s predictive outcomes. To address these issues, this paper introduces a novel end-to-end multi-view decision fusion network specifically designed for remote sensing scene classification. The network integrates information from multi-view remote sensing images under the guidance of image credibility and uncertainty, and when the multi-view image fusion process encounters conflicts, it greatly alleviates the conflicts and provides more reasonable and credible predictions for the multi-view scene classification results. Initially, multi-scale features are extracted from the multi-view images using convolutional neural networks (CNNs). Following this, an asymptotic adaptive feature fusion module (AAFFM) is constructed to gradually integrate these multi-scale features. An adaptive spatial fusion method is then applied to assign different spatial weights to the multi-scale feature maps, thereby significantly enhancing the model’s feature discrimination capability. Finally, an evidence decision fusion module (EDFM), utilizing evidence theory and the Dirichlet distribution, is developed. This module quantitatively assesses the uncertainty in the multi-perspective image classification process. Through the fusing of multi-perspective remote sensing image information in this module, a rational explanation for the prediction results is provided. The efficacy of the proposed method was validated through experiments conducted on the AiRound and CV-BrCT datasets. The results show that our method not only improves single-view scene classification results but also advances multi-view remote sensing scene classification results by accurately characterizing the scene and mitigating the conflicting nature of the fusion process.

Robust HDR reconstruction using 3D patch based on two-scale decomposition

In this paper, we propose an effective 3D patch-based match and fusion method by taking account of dynamic or multi-view scenes in a multi-exposure image sequence using two-scale decomposition. As opposed to most multi-exposure image fusion methods, the proposed method does not require a pre-alignment step to reduce ghosting artifacts. Considering that pixel values are affected by multi-view or multi-exposure scenes, we use a uniform matching approach to match and find similar patches in different exposure images and then fuse them at each scale. By searching all similar patches (instead of the optimal patch) in the searching window to form the 3D patch when facing limited image information, we can fully use the complementary information of the multi-exposure images to preserve information about moving objects and scene details. The experimental results show that the proposed method not only performs well on dynamic scenes but also consistently generates high-quality fused images in multi-view scenes.

Dimensionally Unified Metric Model for Multi-Source and Multi-View Scene Matching

Dimensionally Unified Metric Model for Multi-Source and Multi-View Scene Matching

An end-to-end model for multi-view scene text recognition

Due to the increasing applications of surveillance and monitoring such as person re-identification, vehicle re-identification and sports events tracking, the necessity of text detection and end-to-end recognition is also growing. Although the past deep learning-based models have addressed several challenges such as arbitrary-shaped text, multiple scripts, and variations in the geometric structure of characters, the scope of the models is limited to a single view. This paper presents an end-to-end model for text recognition through refining the multi-views of the same scene, which is called E2EMVSTR (End-to-End Model for Multi-View Scene Text Recognition). Considering the common characteristics shared in multi-view texts, we propose a cycle consistency pairwise similarity-based deep learning model to find texts more efficiently in three input views. Further, the extracted texts are supplied to a Siamese network and semi-supervised attention embedding combinational network for obtaining recognition results. The proposed model combines natural language processing and genetic algorithm models to restore missing character information and correct wrong recognition results. In experiments on our multi-view dataset and several benchmark datasets, the proposed method is proven effective compared to the state-of-the-art methods. The dataset and codes will be made available to the public upon acceptance.

Resolution invariant urban scene classification using Multiview learning paradigm

Urban scene classification is an interesting area in computer vision. The task involves classifying a scene from a pair of aerial-view and ground-view images. Existing approaches have considered single-view and Multiview methods. Multiview approaches have shown to be more robust than single-view. However, most existing Multiview approaches neglect the disparity in the resolution of both views. The aerial-view images are captured with sophisticated high-resolution remote sensing devices. While the ground-view images are captured from closer perspectives with lower resolution. This paper proposed a Multiview scene classification (MuSC) model that caters to the resolution disparity in both views. MuSC introduces a Fourier convolution network (FCN) that is robust to variation of resolution in the cross-view images. The FCN is designed to extract local features (in spatial domain) and global features (in spectral domain). The proposed MuSC has a two-stage classifier. The first stage trains a discriminative view-specific network and classifies each view separately. However, the outputs from each view-specific network are projected into a unified subspace and mutual agreement between them is incentivized through contrastive learning. The second stage integrates the predictions from each view-specific network and trains a unified classifier for final prediction. This integration encourages cross-view complementarity. MuSC is evaluated on two datasets, AiRound and CV-BrCT. Several experiments are conducted with different settings. The results demonstrate that MuSC outperforms existing state-of-the-art models.

F3-Net: Multiview Scene Matching for Drone-Based Geo-Localization

F3-Net: Multiview Scene Matching for Drone-Based Geo-Localization

Implicit Ray Transformers for Multiview Remote Sensing Image Segmentation

The mainstream CNN-based remote sensing (RS) image semantic segmentation approaches typically rely on massively labeled training data. Such a paradigm struggles with the problem of RS multi-view scene segmentation with limited labeled views due to the lack of consideration of 3D information within the scene. In this paper, we propose “Implicit Ray-Transformer (IRT)” based on Implicit Neural Representation (INR) for RS scene semantic segmentation with sparse labels (5% of the images being labeled). We explore a new way of introducing the multi-view 3D structure priors to the task for accurate and view-consistent semantic segmentation. The proposed method includes a two-stage learning process. In the first stage, we optimize a neural field to encode the color and 3D structure of the remote sensing scene based on multi-view images. In the second stage, we design a Ray Transformer to leverage the relations between the neural field 3D features and 2D texture features for learning better semantic representations. Different from previous methods that only consider 3D priors or 2D features, we incorporate additional 2D texture information and 3D priors by broadcasting CNN features to different point features along the sampled ray. To verify the effectiveness of the proposed method, we construct a challenging dataset containing six synthetic sub-datasets collected from the Carla platform and three real sub-datasets from Google Maps. Experiments show that the proposed method outperforms the CNN-based methods and the state-of-the-art INR-based segmentation methods in quantitative and qualitative metrics. The ablation study shows that under a limited number of fully annotated images, the combination of the 3D structure priors and 2D texture can significantly improve the performance and effectively complete missing semantic information in novel views. Experiments also demonstrate the proposed method could yield geometry-consistent segmentation results against illumination changes and viewpoint changes. Our data and code will be public.

Research on the Prediction of the Operational Risk Field of Intelligent Vehicles Based on Dual Multiline LiDAR

To effectively evaluate the risk situation between intelligent vehicles and surrounding traffic participants in complex scenes, a complex traffic environment perception technology based on dual multiline light detection and ranging (LiDAR) is proposed in this work. The vehicle motion state is predicted by fusing the multiview characteristics of point cloud timing and multitarget interaction information, and the risk assessment model is constructed via artificial potential field theory. The real-time point cloud information is used to obtain the time-sequence bird’s-eye view and range image. The improved VGG19 network model is used to extract the time-sequence high-level abstract combined features in the multiview scene. The constructed time-sequence feature vector is used as the input data of the attention mechanism, and the attention-bidirectional long short-term memory (Attention-BiLSTM) model is used for training to form the desired input-output mapping relationship. The motion state of the target vehicle can therefore be updated, and the static and dynamic risk fields of traffic participants surrounding the vehicle can be established based on artificial potential field theory, thereby allowing for the evaluation of the operational risk of the intelligent vehicle. The results of experiments demonstrate that the prediction effect of the target vehicle state parameters via the use of the proposed model is better than that of other compared models, and the prediction effect of the risk field of intelligent vehicle operation based on the multiview point cloud features and vehicle interaction information is good.

Vehicle Motion State Prediction Method Integrating Point Cloud Time Series Multiview Features and Multitarget Interactive Information

A vehicle motion state prediction algorithm integrating point cloud timing multiview features and multitarget interaction information is proposed in this work to effectively predict the motion states of traffic participants around intelligent vehicles in complex scenes. The algorithm analyzes the characteristics of object motion that are affected by the surrounding environment and the interaction of nearby objects and is based on the complex traffic environment perception dual multiline light detection and ranging (LiDAR) technology. The time sequence aerial view map and time sequence front view depth map are obtained using real-time point cloud information perceived by the LiDAR. Time sequence high-level abstract combination features in the multiview scene are then extracted by an improved VGG19 network model and are fused with the potential spatiotemporal interaction of the multitarget operation state data extraction features detected by the laser radar by using a one-dimensional convolution neural network. A temporal feature vector is constructed as the input data of the bidirectional long-term and short-term memory (BiLSTM) network, and the desired input-output mapping relationship is trained to predict the motion state of traffic participants. According to the test results, the proposed BiLSTM model based on point cloud multiview and vehicle interaction information is better than other methods in predicting the state of target vehicles. The results can provide support for the research to evaluate the risk of intelligent vehicle operation environment.

Multi-View Urban Scene Classification with a Complementary-Information Learning Model

Traditional urban scene-classification approaches focus on images taken either by satellite or in aerial view. Although single-view images are able to achieve satisfactory results for scene classification in most situations, the complementary information provided by other image views is needed to further improve performance. Therefore, we present a complementary information-learning model (CILM) to perform multi-view scene classification of aerial and ground-level images. Specifically, the proposed CILM takes aerial and ground-level image pairs as input to learn view-specific features for later fusion to integrate the complementary information. To train CILM, a unified loss consisting of cross entropy and contrastive losses is exploited to force the network to be more robust. Once CILM is trained, the features of each view are extracted via the two proposed feature-extraction scenarios and then fused to train the support vector machine classifier for classification. The experimental results on two publicly available benchmark data sets demonstrate that CILM achieves remarkable performance, indicating that it is an effective model for learning complementary information and thus improving urban scene classification.

Non-aligned multi-view anomaly detection algorithms based on one-cluster clustering

Multi-view data exists widely in reality, and anomalies inevitably accompany it. Existing multi-view anomaly detection methods aim at the aligned multi-view scenes, and they cannot be competent for non-aligned multi-view scenes for that they are more challenging. Therefore, in this paper, we propose four kinds of non-aligned multi-view anomaly detection algorithm based on one-cluster possibilistic C-means (PCM), namely P1M (PCM, where ${\\rm~C}=1$). Different from the aligned multi-view scenes, the non-alignment feature causes the lack of correspondence between the samples in each view, which makes it impossible for us to utilize the local key information at the sample level in the aligned scenes. Therefore, we try to learn from the global information of each view. The key point of this paper is that we try to learn a transformation matrix for each view by means of an alternating optimization method, and project each view into a common subspace. Then the P1M is used to learn the memberships to the target class of all training data in the common subspace, and a threshold of the memberships is set, so as to realize anomaly detection. The experimental results on several UCI datasets show that our proposed method can effectively deal with anomaly detection in the non-aligned multi-view scenes.

Local Feature Performance Evaluation for Structure-From-Motion and Multi-View Stereo Using Simulated City-Scale Aerial Imagery

Ubiquitous low cost multi-rotor and fixed wing drones or unmanned aerial vehicles (UAVs) have accelerated the need for reliable, robust, and scalable Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipelines suitable for a variety of flightpath trajectories especially in degraded environments. Feature tracking being a core part of SfM and MVS, is essential for multiview scene modeling and perception, but difficult to evaluate in large scale datasets due to the lack of sufficient ground-truth. For large-scale aerial imagery, accurate camera orientation and dense 3D point cloud accuracy can be used to assess the impact of accurate feature localization and track length. We propose a novel view simulation (or synthesis) framework which generates visually realistic new unseen camera views for feature detection using known high fidelity camera poses for modeling. Seven state-of-the-art local handcrafted and learning-based features are quantitatively evaluated for robustness and matchability within the SfM and MVS pipelines using the open source COLMAP software. Our experimental results provide performance rankings of each feature, using twelve different evaluation metrics across three synthetic city-wide aerial image sequences. We show that recent learned features, SuperPoint and LF-Net, have not only reached the quality of the best handcrafted features like SIFT and SURF, but now outperform them in terms of more accurate 3D camera pose estimates and longer feature tracks. SuperPoint produces 1.51 meter average position error and 0.03° average angular error, while SIFT remains competitive (second best for pose and overall) with 1.78 meter and 0.11° errors respectively.

AiRound and CV-BrCT: Novel Multiview Datasets for Scene Classification

It is undeniable that aerial/satellite images can provide useful information for a large variety of tasks. But, since these images are always taken from above, some applications can benefit from complementary information provided by other perspective views of the scene, such as ground-level images. Despite a large number of public repositories for both georeferenced photographs and aerial images, there is a lack of benchmark datasets that allow the development of approaches that exploit the benefits and complementarity of aerial/ground imagery. In this article, we present two new publicly available datasets named AiRound and CV-BrCT. The first one contains triplets of images from the same geographic coordinate with different perspectives of view extracted from various places around the world. Each triplet is composed of an aerial RGB image, a ground-level perspective image, and a Sentinel-2 sample. The second dataset contains pairs of aerial and street-level images extracted from southeast Brazil. We design an extensive set of experiments concerning multiview scene classification, using early and late fusion. Such experiments were conducted to show that image classification can be enhanced using multiview data.

View Synthesis from multi-view RGB data using multilayered representation and volumetric estimation

BackgroundAiming at free-view exploration of complicated scenes, this paper presents a method for interpolating views among multi RGB cameras. MethodsIn this study, we combine the idea of cost volume, which represent 3D information, and 2D semantic segmentation of the scene, to accomplish view synthesis of complicated scenes. We use the idea of cost volume to estimate the depth and confidence map of the scene, and use a multi-layer representation and resolution of the data to optimize the view synthesis of the main object. Results/ConclusionsBy applying different treatment methods on different layers of the volume, we can handle complicated scenes containing multiple persons and plentiful occlusions. We also propose the view-interpolation→multi-view reconstruction→view interpolation pipeline to iteratively optimize the result. We test our method on varying data of multi-view scenes and generate decent results.

Text Co-detection in Multi-view Scene.

Multi-view scene analysis has been widely explored in computer vision, including numerous practical applications. The texts in multi-view scenes are often detected by following the existing text detection method in a single image, which however ignores the multi-view corresponding constraint. The multi-view correspondences may contain structure, location information and assist difficulties induced by factors like occlusion and perspective distortion, which are deficient in the single image scene. In this paper, we address the corresponding text detection task and propose a novel text co-detection method to identify the cooccurring texts among multi-view scene images with compositions of detection and correspondence under large environmental variations. In our text co-detection method, the visual and geometrical correspondences are designed to explore texts holding high pairwise representation similarity and guide the exploitation of texts with geometrical correspondences, simultaneously. To guarantee the pairwise consistency among multiple images, we additionally incorporate the cycle consistency constraint, which guarantees alignments of text correspondences in the image set. Finally, text correspondence is represented by a permutation matrix and solved via positive semidefinite and low-rank constraints. Moreover, we also collect a new text co-detection dataset consisting of multi-view image groups obtained from the same scene with different photographing conditions. The experiments show that our text co-detection obtains satisfactory performance and outperforms the related state-of-the-art text detection methods.

Multiview Scene Image Inpainting Based on Conditional Generative Adversarial Networks

With the help of a multiview system, an unmanned vehicle system can better understand the surrounding environment and choose a more accurate and safer path to avoid obstacles. However, due to the interference of the signal or the loss of part of the signal during the acquisition, processing, compression, transmission and decompression of the video image signal, the local area of the image is abnormal, which affects the perception and decision of the system. This article addresses the problems of inaccurate restored images and noise in the restored images by proposing an image restoration method that is applied to a multicamera system. We utilize different perspective images captured by different cameras to assist and constrain the restoration of the damaged image. This method restores the image by combining sample representations and sample distribution models which respectively based on self-encoder reconstruction loss learning and generative adversarial networks. In this method, the infrastructure is a conditional generative adversarial network, the condition is the images that are from the other perspectives, and the generator is a self-encoder structure with cross-layer connection, group convolution and feature map channel exchanged. This method was carried out on a dataset recorded in Zurich using a pair of cameras mounted on a mobile platform. The experimental results demonstrate that the proposed method is superior to the existing methods in terms of mean L1 Loss, mean L2 Loss and the peak signal to noise ratio (PSNR).

A object detection algorithm based on pyramid Convolutional Neural Networks (CNN) and feature map fusion model

Abstract. The aim of this paper is to solve two problems: object detection of small objects and multi-view scenes. First, in practical applications, the collected traffic video is affected by the resolution, viewing angle, focal length and model of the front-end acquisition device. The object size, shape and attitude of the video to be detected are different, resulting in the overall detection performance of the algorithm recognition. In particular, for traffic intersections, the size of the vehicle is related to the distance between the vehicle and the camera, and the object resolution of the vehicle near the intersection is relatively high. As the relative distance increases, the resolution of the object gradually decreases, resulting in feature extraction of the detection object to be detected. And identification becomes more and more difficult, and the probability of the object being detected is greatly reduced. Secondly, there are usually many ways to collect traffic data, such as fixed-position camera, high-altitude camera, and cruising UAV (Unmanned Aerial Vehicle). These video sources collected at different viewing angles and locations pose challenges to the stability, robustness, and generalization capabilities of the detection algorithms. Therefore, design a new algorithm and optimizing model parameters and training samples of different source data is extremely important for multi-view object detection.An object detection algorithm based on pyramid Convolutional Neural Networks (CNN) and feature map fusion method was proposed, and the deep learning technology and the object detection algorithm are used to detect and identify the video objects of multiple viewing angles and different resolution scenes in traffic field. By mixing the lower and deeper feature map model, the algorithm can detect a smaller object in multiple viewing angles and different resolution scenes. Meanwhile, an image block and multi-threading technology was used to avoid scale limit of input image. The experiments show that it can be more efficient and accurate in practical applications of traffic detection filed.The method can be used for the existing network model (VGG16, ResNet101, etc.) to build the skeleton of the object detection algorithm. A new object detection algorithm is developed for these goals, which contain small object recognition and multi-view recognition of traffic video, and it can enable it to extract the lower features of the object and effectively realize multi-object recognition of different scenes. Using the pyramid CNN model, it is possible to effectively combine low-level features and high-level features to achieve feature extraction and fusion of the object, and to solve the problem of small object recognition accuracy to a certain extent. Meanwhile, in view of the shortcomings of the existing object detection algorithm to re-compress the image size, the image block and multi-threading technology are used to restore the original resolution of the image. By using this technology, the accuracy of image object to be detected can be improved.