Multi-view Fusion Research Articles

MVFRnet: A Novel High-Accuracy Network for ISAR Air-Target Recognition via Multi-View Fusion

Inverse Synthetic Aperture Radar (ISAR) is a promising technique for air target imaging and recognition. However, the traditional monostatic ISAR only can provide partial features of the observed target, which is a challenge for high-accuracy recognition. In this paper, to improve the recognition accuracy of air targets, we propose a novel recognition network based on multi-view ISAR imaging and fusion, called Multi-View Fusion Recognition network (MVFRnet). The main structure of MVFRnet consists of two components, the image fusion module and the target recognition module. The fusion module is used for multi-view ISAR data and image preprocessing and mainly performs imaging spatial match, image registration, and weighted fusion. The recognition network consists of the Skip Connect Unit and the Gated Channel Transformation (GCT) attention module, where the Skip Connect Unit ensures the extraction of global depth features of the image and the attention module enhances the perception of shallow contour features of the image. In addition, MVFRnet has a strong perception of image details and suppresses the effect of noise. Finally, simulated and real data are used to verify the effectiveness of the proposed scheme. Multi-view ISAR echoes of six types of aircraft are produced by electromagnetic simulation software. In addition, we also build a millimeter wave ground-based bistatic ISAR experiment system and collect multi-view data from an aircraft model. The simulation and experiment results demonstrate that the proposed scheme can obtain a higher recognition accuracy compared to other state-of-the-art methods. The recognition accuracy can be improved by approximately 30% compared with traditional monostatic recognition.

Multi-view fusion neural network for traffic demand prediction

The extraction of spatial-temporal features is a crucial research in transportation studies, and current studies typically use a unified temporal modeling mechanism and fixed spatial graph for this purpose. However, the fixed spatial graph restricts the extraction of spatial features for similar but not directly connected nodes, while the unified temporal modeling mechanism overlooks the heterogeneity of temporal variation of different nodes. To address these challenges, a multi-view fusion neural network (MVFN) approach is proposed. In this approach, spatial local features are extracted through the use of a graph convolutional network (GCN), and spatial global features are extracted using a cosine re-weighting linear attention mechanism (CLA). The GCN and CLA are combined to create a graph-cosine module (GCM) for the extraction of overall spatial features. Additionally, the multi-channel separable temporal convolutional network (MSTCN) makes use of a multi-channel temporal convolutional network (MTCN) at each layer to extract unified temporal features, and a separable temporal convolutional network (STCN) to extract independent temporal features. Finally, the spatial-temporal feature data is input into the prediction layer to obtain the final result. The model has been validated on two traffic demand datasets and achieved the best prediction accuracy.

Multi-view fusion guided matrix factorization based one-step efficient subspace clustering

Multi-view clustering that integrates the complementary information from different views for better clustering is a fundamental topic in data engineering. Most existing methods learn latent representations first, and then obtain the final result via post-processing. These two-step strategies may lead to sub-optimal clustering. The existing one-step methods are based on spectral clustering, which is inefficient. To address these problems, we propose a Multi-view fusion guided Matrix factorization based One-step subspace Clustering (MMOC) to perform clustering on multi-view data efficiently and effectively in one step. Specifically, we first propose a matrix factorization based multi-view fusion representation method, which adopts efficient matrix factorization instead of time-consuming spectral representation to reduce the computational complexity. Then we propose a self-supervised weight learning strategy to distinguish the importance of different views, which considers both the gradient and the learning rate to make the learned weights closer to the real situation. Finally, we propose a one-step framework of MMOC, which effectively reduces the information loss by integrating data representation, multi-view data fusion, and clustering into one step. We conduct experiments on 5 real-world datasets. The experimental results show the effectiveness and the efficiency of our MMOC method in comparison with state-of-the-art methods.

Semi-Supervised Multi-View Fusion for Identifying CAP and COVID-19 With Unlabeled CT Images

Recently, under the condition of reducing nucleic acid testing for COVID-19 in large population, the computer-aided diagnosis with the chest computed tomography (CT) image has become increasingly important in differential diagnosis of community-acquired pneumonia (CAP) and COVID-19. In practice, there usually exist a mass of unlabeled CT images, especially in regions without adequate medical resources, and the existing diagnosis methods cannot take advantage of the useful information among them. Therefore, it is practical and urgent need to develop a computer-aided diagnosis model that can effectively exploit both labeled and unlabeled samples. To this end, in this paper, we propose a semi-supervised multi-view fusion method for the diagnosis of COVID-19. It explores both the discriminative features from labeled samples and the structure information from unlabeled samples and fuses multi-view features extracted from CT images, including image feature, statistical feature, and lesions specific feature, for improving the diagnostic performance. Specifically, in the proposed model, we utilize semi-supervised learning technique with pairwise constraint regularization to learn the model with both labeled samples and unlabeled samples. Simultaneously, we employ low-rank multi-view constraint to capture latent complementary information among different features from CT images. Experimental results show that the proposed method outperforms the state-of-the-art methods in differential diagnosis of CAP vs. COVID-19.

Multiview Fusion Using Transformer Model for Recommender Systems: Integrating the Utility Matrix and Textual Sources

Recommender systems are challenged with providing accurate recommendations that meet the diverse preferences of users. The main information sources for these systems are the utility matrix and textual sources, such as item descriptions, users’ reviews, and users’ profiles. Incorporating diverse sources of information is a reasonable approach to improving recommendation accuracy. However, most studies primarily use the utility matrix, and when they use textual sources they do not integrate them with the utility matrix. This is due to the risk of combined information causing noise and reducing the effectiveness of good sources. To overcome this challenge, in this study we propose a novel method that utilizes the Transformer Model, a deep learning model that efficiently integrates textual and utility matrix information. The study suggests feature extraction techniques suitable for each information source and an effective integration method in the Transformer model. The experimental results indicate that the proposed model significantly improves recommendation accuracy compared to the baseline model (MLP) for the Mean Absolute Error (MAE) metric, with a reduction range of 10.79% to 31.03% for the Amazon sub-datasets. Furthermore, when compared to SVD, which is known as one of the most efficient models for recommender systems, the proposed model shows a decrease in the MAE metric by a range of 34.82% to 56.17% for the Amazon sub-datasets. Our proposed model also outperforms the graph-based model with an increase of up to 108% in Precision, a decrease of up to 65.37% in MAE, and a decrease of up to 59.24% in RMSE. Additionally, experimental results on the Movielens and Amazon datasets also demonstrate that our proposed model, which combines information from the utility matrix and textual sources, yields better results compared to using only information from the utility matrix.

Multiview Embedding with Partial Labels to Recognize Users of Devices Based on Unified Transformer

Recognizing the users of devices (or clusters of devices) who use IP addresses as unique identities on the Internet can easily enable numerous security applications. Fast and accurate user recognition is critical for supervisors to find influenced organizations connected to their networks in light of new security threats. Many users’ information scatters in the multisource data of IP addresses. Up until now, user recognition of devices has had two main problems. On the one hand, existing methods could not fully use multisource data of the IP addresses and wastes the valuable information of labels. On the other hand, only a tiny portion of devices can be tagged with highly confident known users manually, making it an urgent need to infer unknown users of devices. So, the problem of user recognition on devices is to guess the unknown user with multisource data and existing devices with known users. Therefore, this paper proposes a multiview fusion method to deal with multisource data from devices with a small number of manually labelled samples. The paper uses GraphSAGE to obtain an exemplary representation of IP addresses and designs a label encoder to fully use a small number of devices with known users. Then, the paper builds a specific unified transformer to achieve high performance to determine whether two devices have the same user. At the same time, the paper conducts real-world experiments and finds that the proposed method can achieve 0.9158 accuracy and 0.6131 F1 to find devices with the same users on the constructed dataset in the real world.

Multi-view overlapping clustering for the identification of the subject matter of legal judgments

The legal field is generally burdened by paper-heavy activities, and the management of massive amounts of legal judgments without the adoption of computational tools may compromise the effectiveness and efficiency of administration processes. In this paper, we propose MOSTA, a novel unsupervised method to support the automated identification of groups of legal judgments with similar characteristics, with the goal of reducing the manual effort necessary for the management of legal judgments.Methodologically, MOSTA learns two different embedding models for legal judgments. The first aims to represent the semantics of the textual content, while the second aims to represent co-citations of legal acts, also considering the granularity of the citations. Such representations are then fused through a multi-view approach based on an autoencoder, and the obtained representation is finally exploited by a novel overlapping clustering algorithm. The latter is an additional strong point of MOSTA, since, contrary to existing approaches, does not rely on additional input parameters that inherently influence the degree of overlap of the resulting clusters.Our experiments, performed on three textual datasets, including a real-world legal dataset provided by EUR-Lex, proved that the proposed representation of cited legal acts, the adopted multi-view fusion strategy, and the novel overlapping clustering algorithm implemented in MOSTA provide a positive contribution to the quality of the identified clusters. Finally, MOSTA demonstrated to be able to outperform by a great margin existing complete solutions based on fine-tuned BERT embedding models and existing overlapping clustering algorithms.

A Multi-View Learning Approach for Detecting Personality Disorders Among Arab Social Media Users

Multi-view fusion approaches have gained increasing interest in the past few years by researchers. This interest comes due to the many perspectives that datasets can be looked at and evaluated. One of the most urging areas that require constant leveraging with latest technologies and multi-perspective judgments is the area of psychology. In this article, a novel multi-view fusion model using deep learning algorithms is presented to detect popular types of personality disorders among Arab users of the Twitter platform in an expert-driven fashion, based on the descriptions of the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition. To the best of our knowledge, the work presented is the first of its kind with no publicly available datasets that report statements around personality disorders in the Arabic language, and thus we created AraPerson, a dataset that consists of 8,000 textual tweets coupled with 8,000 images that prescribe mental statuses for a total of 150 users collected with regular expressions generated under the supervision of domain experts. The dataset was fed into a baseline multi-view model by combining a CNN model with a Bi-LSTM model to detect two types of popular personality disorders by analyzing textual and visual posts on 150 user profiles. The experiments were followed with fusing the DenseNet model with the Bi-LSTM model, experimenting with the effect of using concatenation, addition, and multiplication methods for vectors’ combination. The work presented in this article is unprecedented, specifically in a controversial area such as personality disorders detection among Arab communities. The best reported accuracy is 87%, which is quite promising, as the two types of personality disorders investigated are highly overlapping.

Multi-view rotating machinery fault diagnosis with adaptive co-attention fusion network

Intelligent fault diagnosis is an intriguing topic, attracting increasing interest in safe and reliable industrial production. Tremendous progress has been made in recent years in developing better fault diagnosis methods. Nevertheless, most methods rely on an individual vibration signal while ignoring the consensus and complementary between different views of the signal. Towards this end, we propose a novel method named COFU, i.e., a multi-view learning model with CO-attention FUsion network for rotating machinery fault diagnosis, which primarily exploits consensus and complementary across multiple views. Specifically, we first utilize three different encoders to construct high-level feature spaces of multiple views. Then the adaptive co-attention fusion network is designed to learn an integrated representation where rich associations among these feature spaces are fully considered. Finally, the fault detector fed by the fused representation is devised to diagnose the fault category. To affirm the efficacy of the proposed approach, a comprehensive evaluation has been conducted on the CWRU, SEU_bearing, and SEU_gear datasets. The results indicate that the accuracy of the COFU method is 100%, 99.95%, and 100%, respectively. Encouraging findings demonstrate that our method outperforms all the baseline methods. Furthermore, it is observed that the COFU method demonstrates improved performance when applied in noisy environments. This study offers a promising solution that ensures the great potential of multi-view fusion in rotating machinery fault diagnosis.

Dual-View Single-Shot Multibox Detector at Urban Intersections: Settings and Performance Evaluation

The explosion of artificial intelligence methods has paved the way for more sophisticated smart mobility solutions. In this work, we present a multi-camera video content analysis (VCA) system that exploits a single-shot multibox detector (SSD) network to detect vehicles, riders, and pedestrians and triggers alerts to drivers of public transportation vehicles approaching the surveilled area. The evaluation of the VCA system will address both detection and alert generation performance by combining visual and quantitative approaches. Starting from a SSD model trained for a single camera, we added a second one, under a different field of view (FOV) to improve the accuracy and reliability of the system. Due to real-time constraints, the complexity of the VCA system must be limited, thus calling for a simple multi-view fusion method. According to the experimental test-bed, the use of two cameras achieves a better balance between precision (68%) and recall (84%) with respect to the use of a single camera (i.e., 62% precision and 86% recall). In addition, a system evaluation in temporal terms is provided, showing that missed alerts (false negatives) and wrong alerts (false positives) are typically transitory events. Therefore, adding spatial and temporal redundancy increases the overall reliability of the VCA system.

Multiview-Learning-Based Generic Palmprint Recognition: A Literature Review

Palmprint recognition has been widely applied to security authentication due to its rich characteristics, i.e., local direction, wrinkle, and texture. However, different types of palmprint images captured from different application scenarios usually contain a variety of dominant features. Specifically, the palmprint recognition performance will be degraded by the interference factors, i.e., noise, rotations, and shadows, while palmprint images are acquired in the open-set environments. Seeking to handle the long-standing interference information in the images, multiview palmprint feature learning has been proposed to enhance the feature expression by exploiting multiple characteristics from diverse views. In this paper, we first introduced six types of palmprint representation methods published from 2004 to 2022, which described the characteristics of palmprints from a single view. Afterward, a number of multiview-learning-based palmprint recognition methods (2004–2022) were listed, which discussed how to achieve better recognition performances by adopting different complementary types of features from multiple views. To date, there is no work to summarize the multiview fusion for different types of palmprint features. In this paper, the aims, frameworks, and related methods of multiview palmprint representation will be summarized in detail.

Learning multi-view and centerline topology connectivity information for pulmonary artery–vein separation

Background:Automatic pulmonary artery–vein separation has considerable importance in the diagnosis and treatment of lung diseases. However, insufficient connectivity and spatial inconsistency have always been the problems of artery–vein separation. Methods:A novel automatic method for artery–vein separation in CT images is presented in this work. Specifically, a multi-scale information aggregated network (MSIA-Net) including multi-scale fusion blocks and deep supervision, is proposed to learn the features of artery–vein and aggregate additional semantic information, respectively. The proposed method integrates nine MSIA-Net models for artery–vein separation, vessel segmentation, and centerline separation tasks along with axial, coronal, and sagittal multi-view slices. First, the preliminary artery–vein separation results are obtained by the proposed multi-view fusion strategy (MVFS). Then, centerline correction algorithm (CCA) is used to correct the preliminary results of artery–vein separation by the centerline separation results. Finally, the vessel segmentation results are utilized to reconstruct the artery–vein morphology. In addition, weighted cross-entropy and dice loss are employed to solve the class imbalance problem. Results:We constructed 50 manually labeled contrast-enhanced computed CT scans for five-fold cross-validation, and experimental results demonstrated that our method achieves superior segmentation performance of 97.7%, 85.1%, and 84.9% on ACC, Pre, and DSC, respectively. Additionally, a series of ablation studies demonstrate the effectiveness of the proposed components. Conclusion:The proposed method can effectively solve the problem of insufficient vascular connectivity and correct the spatial inconsistency of artery–vein.

Transformer-based comparative multi-view illegal transaction detection.

In recent years, as the Ether platform has grown by leaps and bounds. Numerous unscrupulous individuals have used illegal transaction to defraud large sums of money, causing billions of dollars of losses to investors worldwide. Facing the endless stream of the illegal transaction based on Ether smart contracts problems, such as illegal transaction, money laundering, financial fraud, phishing. Currently, illegal transaction are only detected by a single view of the smart contract's contract code view feature and account transaction view feature, which is not only incomplete, but also not fully representative of the smart contract's features. More importantly, the single view detection model cannot accurately capture the global structure and semantic features between the Tokens of the view features. In this case, it is particularly important that all view features are shared among themselves. In this paper, we investigate a Transformer-based model for contrasting illegal transaction detection networks under multiple views (TranMulti-View Net). The model in this paper is based on Transformer to learn a multi-view fusion representation, which aims to maximise the fusion of the interaction information of different view features under the same condition. In this model we first use the Transformer model to learn global structure and semantic features from a sequence of Tokens tokenised by a view, capturing the remote dependencies of Tokens in the view features, and then we share the contract code view features and the account transaction view features across all views to learn important semantic information between views from each other. In addition, we find that the approach of semi-supervised training of multi-view features using contrast learning outperforms the scheme of prediction based on direct fusion of different view features, resulting in stronger correlation between view features. As a result, the underlying semantic information can be captured more accurately, leading to more accurate predictions of illegal transaction. The experimental results show that our proposed TranMulti-View Net obtains good detection results with a Precision score of 98%.

3D Reconstruction for Multi-view Objects

Deep learning-based 3D reconstruction neural networks have achieved good performance on generating 3D features from 2D features. However, they often lead to feature loss in reconstruction. In this paper we propose a multi-view object 3D reconstruction neural network, named P2VNet. The depth estimation module of the front and back layers of P2VNet realizes the smooth transformation from 2D features to 3D features, which improves the performance of single view reconstruction. A multi-scale fusion sensing module in multi-view fusion is also proposed, where more receptive fields are added to generate richer context-aware features. We also introduce 3DFocal Loss to replace binary cross-entropy to address the problems of unbalanced space occupation of the voxel grid and complex division of partial grid occupation. Our experimental results have demonstrated that P2VNet has achieved higher accuracy than existing works.

Exploring novel disease-disease associations based on multi-view fusion network

Established taxonomy system based on disease symptom and tissue characteristics have provided an important basis for physicians to correctly identify diseases and treat them successfully. However, these classifications tend to be based on phenotypic observations, lacking a molecular biological foundation. Therefore, there is an urgent to integrate multi-dimensional molecular biological information or multi-omics data to redefine disease classification in order to provide a powerful perspective for understanding the molecular structure of diseases. Therefore, we offer a flexible disease classification that integrates the biological process, gene expression, and symptom phenotype of diseases, and propose a disease-disease association network based on multi-view fusion. We applied the fusion approach to 223 diseases and divided them into 24 disease clusters. The contribution of internal and external edges of disease clusters were analyzed. The results of the fusion model were compared with Medical Subject Headings, a traditional and commonly used disease taxonomy. Then, experimental results of model performance comparison show that our approach performs better than other integration methods. As it was observed, the obtained clusters provided more interesting and novel disease-disease associations. This multi-view human disease association network describes relationships between diseases based on multiple molecular levels, thus breaking through the limitation of the disease classification system based on tissues and organs. This approach which motivates clinicians and researchers to reposition the understanding of diseases and explore diagnosis and therapy strategies, extends the existing disease taxonomy. Availability of data and materialsThe preprocessed dataset and source code supporting the conclusions of this article are available at GitHub repository https://github.com/yangxiaoxi89/mvHDN.

Incomplete Multiview Clustering via Cross-View Relation Transfer

In this paper, we consider the problem of multi-view clustering on incomplete views. Compared with complete multi-view clustering, the view-missing problem increases the difficulty of learning common representations from different views. To address the challenge, we propose a novel incomplete multi-view clustering framework, which incorporates cross-view relation transfer and multi-view fusion learning. Specifically, based on the consistency existing in multi-view data, we devise a cross-view relation transfer-based completion module, which transfers known similar inter-instance relationships to the missing view and recovers the missing data via graph networks based on the transferred relationship graph. Then the view-specific encoders are designed to extract the recovered multi-view data, and an attention-based fusion layer is introduced to obtain the common representation. Moreover, to reduce the impact of the error caused by the inconsistency between views and obtain a better clustering structure, a joint clustering layer is introduced to optimize recovery and clustering simultaneously. Extensive experiments conducted on several real datasets demonstrate the effectiveness of the proposed method.

Suspicious Object Detection for Millimeter-Wave Images with Multi-View Fusion Siamese Network.

Millimeter-wave (MMW) imaging techniques have been widely used in the public security industries for their under-controlled privacy concerns and no health hazards. However, since MMW images are low resolution and most objects are small, reflection-weak, diverse, suspicious object detection in the MMW images is a very challenging task. This paper develops a robust suspicious object detector for the MMW images based on the Siamese network integrated with the pose estimation and image segmentation, which estimates the coordinates of human joints and segments the complete human images into symmetrical body part images. Unlike most existing detectors, which detect and recognize suspicious objects in MMW images and require a complete training set with correct annotations, our proposed model aims to learn the similarity between two symmetrical human body part images segmented from the complete MMW images. Furthermore, to decrease the misdetection caused by the restricted field of view, we further fuse the multi-view MMW images observed from the same person by designing a decision-level fusion strategy and feature-level fusion strategy based on the attention mechanism. Experimental results on the measured MMW images show that our proposed models have favorable detection accuracy and speed in practical application and thus prove their effectiveness.

A multi-view attention-based spatial–temporal network for airport arrival flow prediction

Accurate airport arrival flow prediction is a precondition for intelligent air traffic flow management. However, most existing studies focus on the dynamic traffic flow in a single-airport scenario, which usually ignores the spatial interactions among airports. Modelling network-wide spatial dependencies among airports is difficult because it requires models to consider multiple underlying factors jointly. We propose a multi-view fusion approach to automatically learn an adjacency matrix from flight duration and flight schedule factors. The learned adjacency matrix is then fed into a specially designed graph convolutional block, which governs the message passing process among airports. Finally, the graph convolutional block with the learned adjacency matrix is embedded into the gated recurrent units to capture temporal dependencies. Experimental results on a real-world dataset for the multistep prediction task show the effectiveness and efficacy of the proposed model. In addition, visualisation and analysis of the learned adjacency matrix verify that the proposed multi-view fusion approach is capable of learning informative spatial interaction patterns.

LDI-MVFNet: A Multi-view fusion deep network for leachate distribution imaging

The accurate monitoring and early warning of groundwater pollution caused by the concealed leakage of landfills is a major challenge globally in the field of solid waste management and groundwater protection. Electrical resistivity tomography (ERT) represents a potential solution with advantages, owing to its fast and nondestructive characteristics. However, traditional ERT based on a single array cannot reveal the distribution and dynamics of pollution in complex underground media owing to the limited information it carries. We designed a novel deep network for multi-view fusion to invert the real resistivity distribution of the medium caused by leachate (named LDI-MVFNet) so as to infer the distribution of leachate. To support model establishment and validation, ERT instances collected from synthetic models and a salt tracer experiment were inverted. We compared the inversion results of the LDI-MVFNet with those of single arrays and found that the LDI-MVFNet performed the best overall. The average root mean square error (RMSE) of synthetic models reached 0.98, performing better than Dipole–Dipole (3.86), Wenner–Schlumberger (3.37), and Pole–Pole (6.61), which were inverted separately. The resultant inverted subsurface true resistivity data were presented in the form of two-dimensional (2D) cross sections. The imaging results of 2D cross sections showed that LDI-MVFNet was superior to others in data noise suppression and inversion accuracy. The results of this study indicate that the data fusion of multiple views can more accurately reflect the real resistivity than the inversion of a single array can.

A Belief Two-Level Weighted Clustering Method for Incomplete Pattern Based on Multiview Fusion.

Incomplete pattern clustering is a challenging task because the unknown attributes of the missing data introduce uncertain information that affects the accuracy of the results. In addition, the clustering method based on the single view ignores the complementary information from multiple views. Therefore, a new belief two-level weighted clustering method based on multiview fusion (BTC-MV) is proposed to deal with incomplete patterns. Initially, the BTC-MV method estimates the missing data by an attribute-level weighted imputation method with k-nearest neighbor (KNN) strategy based on multiple views. The unknown attributes are replaced by the average of the KNN. Then, the clustering method based on multiple views is proposed for a complete data set with estimations; the view weights represent the reliability of the evidence from different source spaces. The membership values from multiple views, which indicate the probability of the pattern belonging to different categories, reduce the risk of misclustering. Finally, a view-level weighted fusion strategy based on the belief function theory is proposed to integrate the membership values from different source spaces, which improves the accuracy of the clustering task. To validate the performance of the BTC-MV method, extensive experiments are conducted to compare with classical methods, such as MI-KM, MI-KMVC, KNNI-FCM, and KNNI-MFCM. Results on six UCI data sets show that the error rate of the BTC-MV method is lower than that of the other methods. Therefore, it can be concluded that the BTC-MV method has superior performance in dealing with incomplete patterns.