Multi-view Data Fusion Research Articles

Multi-view clustering via latent consistency multi-graph fusion

Multi-view clustering (MVC) algorithms are prevalent in machine learning tasks due to their superiority in mining complementary information from multi-view data. Despite their widespread use, existing MVC approaches ignore the presence of inconsistencies in the multi-view features during multi-view data fusion, which leads to suboptimal clustering performance. To address this challenge, the paper presents a novel approach called multi-view clustering via latent consistency multi-graph fusion. Specifically, a novel latent representation learning model is devised to learn a unified representation from multi-view features, simultaneously capturing the high-order relationships within the multi-view data. Additionally, to alleviate the adverse effects caused by the inconsistency multi-view feature, a consistent multi-graph fusion module is proposed to provide consistent constraints for latent representations. To be specific, the graphs that reflect the manifold structure for each view are fused to guide a consistent latent representation learning. After that, an alternating optimization algorithm is designed to update variables iteratively and alternatively, among which the fusion weights are assigned adaptively. Finally, comprehensive experiments conducted on seven widely recognized benchmark multi-view datasets affirm the efficacy of the proposed method, showcasing its superiority over state-of-the-art approaches.

End-to-end multiview fusion for building mapping from aerial images

In the domain of photogrammetry, the fusion of information from multiple views holds the potential to significantly enhance the accuracy and robustness of building mapping. While multiview observation and stereoscopic imaging form the bedrock of photogrammetric projects, current deep learning methodologies predominantly focus on orthophotos and digital surface models (DSMs), often sidelining the rich multiview information inherent in original images. Addressing this gap, we present Multiview Mapper (MVMapper), an end-to-end learning framework explicitly crafted to harness and fuse the rich semantic information from original multiview images with object-space features. MVMapper uses stereo labels for supervised building segmentation in a dual-space, encompassing both image and object domains. Additionally, it incorporates a novel piecewise affine projection method for ensuring a robust image-to-object feature transformation. Experimental results on an aerial photogrammetric dataset with a resolution of 30 cm demonstrate MVMapper’s superiority over state-of-the-art multiview data fusion methods, yielding significant improvements in segmentation and contour accuracy. Notably, the proposed piecewise affine projection method mitigates misalignment issues caused by DSM noise, enabling the effective fusion of multiview features with object-space features. Further experimentation on a separate open-source dataset demonstrates MVMapper’s substantial advantages in transferability to other contexts.

Improving 3-D Zebrafish Tracking With Multiview Data Fusion and Global Association

Improving 3-D Zebrafish Tracking With Multiview Data Fusion and Global Association

Multi-view fusion guided matrix factorization based one-step efficient subspace clustering

Multi-view clustering that integrates the complementary information from different views for better clustering is a fundamental topic in data engineering. Most existing methods learn latent representations first, and then obtain the final result via post-processing. These two-step strategies may lead to sub-optimal clustering. The existing one-step methods are based on spectral clustering, which is inefficient. To address these problems, we propose a Multi-view fusion guided Matrix factorization based One-step subspace Clustering (MMOC) to perform clustering on multi-view data efficiently and effectively in one step. Specifically, we first propose a matrix factorization based multi-view fusion representation method, which adopts efficient matrix factorization instead of time-consuming spectral representation to reduce the computational complexity. Then we propose a self-supervised weight learning strategy to distinguish the importance of different views, which considers both the gradient and the learning rate to make the learned weights closer to the real situation. Finally, we propose a one-step framework of MMOC, which effectively reduces the information loss by integrating data representation, multi-view data fusion, and clustering into one step. We conduct experiments on 5 real-world datasets. The experimental results show the effectiveness and the efficiency of our MMOC method in comparison with state-of-the-art methods.

A Pre-Seismic Anomaly Detection Approach Based on Earthquake Cross Partial Multi-View Data Fusion

It is a challenge to detect pre-seismic anomalies by using only one dataset due to the complexity of earthquakes. Therefore, it is a promising direction to use multiparameteric data. The earthquake cross partial multi-view data fusion approach (EQ-CPM) is proposed in this paper. By using this method, electromagnetic data and seismicity indicators are fused. This approach tolerates the absence of data and complements the missing part in fusion. First, the effectiveness of seismicity indicators and electromagnetic data was validated through two earthquake case studies. Then, four machine learning algorithms were applied to detect pre-seismic anomalies by using the fused data and two original datasets. The results show that the fused data provided better performance than the single-modal data. In the Matthews correlation coefficient index, the results of our method showed an 8% improvement compared with the latest study.

Label recovery and label correlation co-learning for multi-view multi-label classification with incomplete labels.

Multi-view multi-label learning (MVML) is an important paradigm in machine learning, where each instance is represented by several heterogeneous views and associated with a set of class labels. However, label incompleteness and the ignorance of both the relationships among views and the correlations among labels will cause performance degradation in MVML algorithms. Accordingly, a novel method, label recovery and label correlation co-learning for Multi-View Multi-Label classification with incoMplete Labels (MV2ML), is proposed in this paper. First, a label correlation-guided binary classifier kernel-based is constructed for each label. Then, we adopt the multi-kernel fusion method to effectively fuse the multi-view data by utilizing the individual and complementary information among multiple views and distinguishing the contribution difference of each view. Finally, we propose a collaborative learning strategy that considers the exploitation of asymmetric label correlations, the fusion of multi-view data, the recovery of incomplete label matrix and the construction of the classification model simultaneously. In such a way, the recovery of incomplete label matrix and the learning of label correlations interact and boost each other to guide the training of classifiers. Extensive experimental results demonstrate that MV2ML achieves highly competitive classification performance against state-of-the-art approaches on various real-world multi-view multi-label datasets in terms of six evaluation criteria.

Trusted Multi-View Deep Learning with Opinion Aggregation

Multi-view deep learning is performed based on the deep fusion of data from multiple sources, i.e. data with multiple views. However, due to the property differences and inconsistency of data sources, the deep learning results based on the fusion of multi-view data may be uncertain and unreliable. It is required to reduce the uncertainty in data fusion and implement the trusted multi-view deep learning. Aiming at the problem, we revisit the multi-view learning from the perspective of opinion aggregation and thereby devise a trusted multi-view deep learning method. Within this method, we adopt evidence theory to formulate the uncertainty of opinions as learning results from different data sources and measure the uncertainty of opinion aggregation as multi-view learning results through evidence accumulation. We prove that accumulating the evidences from multiple data views will decrease the uncertainty in multi-view deep learning and facilitate to achieve the trusted learning results. Experiments on various kinds of multi-view datasets verify the reliability and robustness of the proposed multi-view deep learning method.

Multi-view data fusion in multi-object tracking with probability density-based ordered weighted aggregation

In this paper, a method is presented for Multi-Object Tracking (MOT) in presence of partial or complete occlusions. This work focuses on improved object detection and data association in a single view, and also fuses data from multiple views using the Ordered Weighted Aggregation (OWA) algorithm. Hence, a deep learning model was proposed to detect objects more accurately in a tracking-by-detection framework. This paper aims to enhance object detection, data association, and the trajectories of the objects in the MOT algorithm respectively by applying Mask R-CNN, Zernike Moments and combination of several similarity metrics, and fusion of multi-camera information by probability density-based OWA (PD-OWA). The inter-frame detected objects are matched based on the appropriate similarity metrics. In the fusion part, the Kernel Density Estimation (KDE) is utilized to assign weight coefficient to each camera view and determine the descending order of data in the OWA algorithm. Finally, the positions of each object coming from different views are weighted and aggregated. The results show that the proposed method improves object detection, association performance, and tracking trajectory in the “PETS09-S2L1″ and the “EPFL Terrace” video sequences and achieved 81.6% and 79.6% multiple objects tracking accuracy (MOTA), respectively.

Motion parameters measurement of user-defined key points using 3D pose estimation

Motion parameters measurement is essential for understanding animal behavior, exploring the laws of object motion, and studying control methods. Nowadays, advanced computer vision based on machine learning technology supports markerless object tracking in 2D videos. However, due to the fact that all objects move in three-dimensional space, this paper introduces a method of measuring motion parameters using 3D pose estimation. First, an enhanced iterative bundle adjustment algorithm is proposed for multi-camera calibration in a multi-camera vision system by adding two control parameters, which dramatically reduces the reprojection error of multi-camera calibration and lays the foundation for high-precision triangulation. Then, a new spatiotemporal loss function is proposed, which considers the relationship between key points that do not constitute limbs, thereby improving triangulation accuracy. The new multi-camera calibration algorithm is evaluated on ChArUco and 3D pose estimation for metronome, planet pendulum, human hand, Koi, and cheetah. The experimental results show that: (1) the two hyper-parameters in the enhanced iterative bundle adjustment algorithm effectively suppress the influence of noise and play a good role in reducing the reprojection error of multi-camera calibration; (2) the spatiotemporal loss function has a strong constraining ability, the time loss can stabilize high frame rate video triangulation to maintain accuracy, while the space loss can improve the accuracy of triangulation for more complex structures; (3) multi-view data fusion is also conducive to improving the accuracy of triangulation. Moreover, the method was successfully applied to some actual measurement scenes: (1) the accurate measurement of the frequency of a metronome; and (2) the success measurement of the movement of a Koi, which conforms to the basic model of fish swimming. Some dynamic measurement results are displayed at https://github.com/wux024/AdamPose.

Evaluating the Street Greening with the Multiview Data Fusion

Street greening, an indispensable element of urban green spaces, has played an important role in beautifying the environment, alleviating the urban heat island effect, and improving residents’ comfort. Vegetation coverage is a common index used for measuring street greening. However, there are some shortcomings in the traditional evaluation methods of vegetation coverage. Part of the vegetation coverage cannot be determined from a two‐dimensional perspective, such as shrubs and green walls. In this paper, the Sentinel‐2 image was used to extract the street fractional vegetation cover (SFVC) and the Baidu street view panoramas were used to extract the green view index (GVI). To overcome the lack of a single perspective from the street vegetation coverage evaluation, the above two indices were merged to construct a comprehensive street greening evaluation index (CSGEI). The research area is the Longhua District of Shenzhen city in Southern China. All three indices were divided into five classes using natural breakpoint methods based on previous research experience. The results showed that Baidu street view panoramas could effectively identify shrubs and green walls that were deficient in the Sentinel‐2 image. The GVI is a supplement to the street vegetation coverage. The SFVC and GVI were divided into five classes, from L1 to L5 implying a gradual increase in the percentage of the vegetated area. The result has shown that the SFVC was in the L1, accounting for 53.68%. After index merging, the process of accounting for the L1 decreased to 31.29%. The multiperspective integrated CSGEI could comprehensively measure the distribution information of street greening and guide the planning and management of urban green landscapes.

Adaptive graph weighting for multi-view dimensionality reduction

Multi-view learning has become a flourishing topic in recent years since it can discover various informative structures with respect to disparate statistical properties. However, multi-view data fusion remains challenging when exploring a proper way to find shared while complementary information. In this paper, we present an adaptive graph weighting scheme to conduct semi-supervised multi-view dimensional reduction. Particularly, we construct a Laplacian graph for each view, and thus the final graph is approximately regarded as a centroid of these single view graphs with different weights. Based on the learned graph, a simple yet effective linear regression function is employed to project data into a low-dimensional space. In addition, our proposed scheme can be well extended to an unsupervised version within a unified framework. Extensive experiments on varying benchmark datasets illustrate that our proposed scheme is superior to several state-of-the-art semi-supervised/unsupervised multi-view dimensionality reduction methods. Last but not least, we demonstrate that our proposed scheme provides a unified view to explain and understand a family of traditional schemes.

Self-adaptive morphable model based collaborative multi-view 3d face reconstruction in visual sensor network

3D face reconstruction is an efficient method for pedestrian recognition in non-cooperative environment because of its outstanding performance in robust face recognition for uncontrolled pose and illumination changes. Visual sensor network is widely used in target surveillance as powerful unattended distributed measurement systems. This paper proposes a collaborative multi-view non-cooperative 3D face reconstruction method in visual sensor network. A peer-to-peer paradigm-based visual sensor network is employed for distributed pedestrian tracking and optimal face image acquisition. Gaussian probability distribution-based multi-view data fusion is used for target localization, and kalman filter is applied for target tracking. A lightweight face image quality evaluation method is presented to search optimal face images. A self-adaptive morphable model is designed for multiview 3D face reconstruction. To adjust the self-adaptive morphable model, the optimal face images and their poses estimation are used. Cooperative chaotic particle swarm optimization is employed for parameters optimization of the self-adaptive morphable model. Experimental results on real data show that the proposed method can acquire optimal face images and achieve non-cooperative 3D reconstruction efficiently.

Multi-view ensemble manifold regularization for 3D object recognition

View-based methods are popular in 3D object recognition. However, current methods with traditional classifiers are usually based on one-to-one view matching and fail to capture the structure information of multiple views. Some multi-view based methods take different views into consideration, but they still treat views separately. In this paper, we propose a novel 3D object recognizing method based on multi-view data fusion, called Multi-view Ensemble Manifold Regularization (MEMR). In this method, we model image features with a regularization term for SVM. To train this modified SVM, multi-view learning is achieved with alternating optimization. Hypergraph construction is used to better capture the connectivity among views. Experimental results show that the accuracy rate has been improved by 20–25%, which demonstrates the effectiveness of the proposed method.

Real-time multiview data fusion for object tracking with RGBD sensors

SUMMARYThis paper presents a new approach to accurately track a moving vehicle with a multiview setup of red–green–blue depth (RGBD) cameras. We first propose a correction method to eliminate a shift, which occurs in depth sensors when they become worn. This issue could not be otherwise corrected with the ordinary calibration procedure. Next, we present a sensor-wise filtering system to correct for an unknown vehicle motion. A data fusion algorithm is then used to optimally merge the sensor-wise estimated trajectories. We implement most parts of our solution in the graphic processor. Hence, the whole system is able to operate at up to 25 frames per second with a configuration of five cameras. Test results show the accuracy we achieved and the robustness of our solution to overcome uncertainties in the measurements and the modelling.

Multiview light-sheet microscope for rapid in toto imaging

We present a multiview selective-plane illumination microscope (MuVi-SPIM), comprising two detection and illumination objective lenses, that allows rapid in toto fluorescence imaging of biological specimens with subcellular resolution. The fixed geometrical arrangement of the imaging branches enables multiview data fusion in real time. The high speed of MuVi-SPIM allows faithful tracking of nuclei and cell shape changes, which we demonstrate through in toto imaging of the embryonic development of Drosophila melanogaster.

Fast Automatic Matching Method on Multi-View Point Clouds for Rail Tanker

In this paper, a novel method is proposed for automatically matching the measured data of a rail tanker. The spindle orientation of point cloud is determined by PCA, and then any relative point pair is well matched according to the spatial topology relations as the complexity reduction point of view, finally the data is mapped to a uniform coordinate system with the Quaternion method. The experimental results demonstrated that the proposed method achieves simplicity of implementation with robustness, and especially suitable for positioning the work piece of the rail tanker and the multi-view data fusion.