Multi-view Embedding Research Articles

Multi-view visual semantic embedding for cross-modal image–text retrieval

Visual Semantic Embedding (VSE) is a dominant method for cross-modal image–text retrieval. The purpose of VSE is to learn an embedding space where images can be embedded close to the corresponding captions. However, there are large intra-class variations in image–text data. Multiple captions describing the same image may be described from different views, and descriptions of different views are often dissimilar. The VSE method embeds samples from the same class in similar positions, which suppresses intra-class variations and leads to inferior generalization. This paper proposes a Multi-View Visual Semantic Embedding (MV-VSE) framework that learns multiple embeddings for an image, explicitly modeling intra-class variation. To optimize the MV-VSE framework, a multi-view triplet loss is proposed, which jointly optimizes multi-view embeddings while retaining intra-class variation. Recently, large-scale Vision-Language Pre-training (VLP) has become a new paradigm for cross-modal image–text retrieval. To allow our framework to be flexibly applied to the traditional VSE models and VSE-based VLP models, we incorporate the contrastive loss commonly used in VLP and the triplet loss into a unified loss, and further propose a multi-view unified loss. Our framework can be applied plug-and-play to traditional VSE models and VSE-based VLP models without excessively increasing model complexity. Experimental results on the image–text retrieval benchmark datasets demonstrate that applying our framework can boost the retrieval performance of current VSE models. The code is available at https://github.com/AAA-Zheng/MV-VSE.

Prediction of the cutting forces in milling operation based on multi-objective optimization of the time series analysis parameters

Accurate simulation of the cutting forces in the milling process is extremely complicated because of variations in cutting process parameters, including the change in modal characteristics of the workpiece along the machining path and the high-temperature gradient around the cutting zone, especially for the modeling of milling forces in flexible thin-walled workpieces. Therefore, proposing effective methods for prediction and control of cutting forces is highly important to ensure the high performance and stability of the cutting process as well as the quality of the machined surface. In this article, a novel method is presented for rapid prediction of cutting force signals based on the combination of the multi-variable time series analysis, the Imperialist Competitive Algorithm (ICA) and Multi-View Embedding (MVE) algorithm. A state-space model of the dynamic system is presented in order to quickly predict the instantaneous dynamic characteristics of the cutting process. The maximum error between the amplitude of experimental and predicted force signals (peak-to-peak) in stable and unstable cutting conditions is less than 8% and 17%, respectively. Therefore, it is illustrated that the presented method has remarkable computational accuracy and convergence speed for predicting the future of the cutting forces under both stable and unstable conditions.

Graph Contrastive Multi-view Learning: A Pre-training Framework for Graph Classification

Recent advancements in node and graph classification tasks can be attributed to the implementation of contrastive learning and similarity search. Despite considerable progress, these approaches present challenges. The integration of similarity search introduces an additional layer of complexity to the model. At the same time, applying contrastive learning to non-transferable domains or out-of-domain datasets results in less competitive outcomes. In this work, we propose maintaining domain specificity for these tasks, which has demonstrated the potential to improve performance by eliminating the need for additional similarity searches. We adopt a fraction of domain-specific datasets for pre-training purposes, generating augmented pairs that retain structural similarity to the original graph, thereby broadening the number of views. This strategy involves a comprehensive exploration of optimal augmentations to devise multi-view embeddings. An evaluation protocol, which focuses on error minimization, accuracy enhancement, and overfitting prevention, guides this process to learn inherent, transferable structural representations that span diverse datasets. We combine pre-trained embeddings and the source graph as a beneficial input, leveraging local and global graph information to enrich downstream tasks. Furthermore, to maximize the utility of negative samples in contrastive learning, we extend the training mechanism during the pre-training stage. Our method consistently outperforms comparative baseline approaches in comprehensive experiments conducted on benchmark graph datasets of varying sizes and characteristics, establishing new state-of-the-art results.

HGCLAMIR: Hypergraph contrastive learning with attention mechanism and integrated multi-view representation for predicting miRNA-disease associations.

Existing studies have shown that the abnormal expression of microRNAs (miRNAs) usually leads to the occurrence and development of human diseases. Identifying disease-related miRNAs contributes to studying the pathogenesis of diseases at the molecular level. As traditional biological experiments are time-consuming and expensive, computational methods have been used as an effective complement to infer the potential associations between miRNAs and diseases. However, most of the existing computational methods still face three main challenges: (i) learning of high-order relations; (ii) insufficient representation learning ability; (iii) importance learning and integration of multi-view embedding representation. To this end, we developed a HyperGraph Contrastive Learning with view-aware Attention Mechanism and Integrated multi-view Representation (HGCLAMIR) model to discover potential miRNA-disease associations. First, hypergraph convolutional network (HGCN) was utilized to capture high-order complex relations from hypergraphs related to miRNAs and diseases. Then, we combined HGCN with contrastive learning to improve and enhance the embedded representation learning ability of HGCN. Moreover, we introduced view-aware attention mechanism to adaptively weight the embedded representations of different views, thereby obtaining the importance of multi-view latent representations. Next, we innovatively proposed integrated representation learning to integrate the embedded representation information of multiple views for obtaining more reasonable embedding information. Finally, the integrated representation information was fed into a neural network-based matrix completion method to perform miRNA-disease association prediction. Experimental results on the cross-validation set and independent test set indicated that HGCLAMIR can achieve better prediction performance than other baseline models. Furthermore, the results of case studies and enrichment analysis further demonstrated the accuracy of HGCLAMIR and unconfirmed potential associations had biological significance.

Multiview adaptive attention pooling for image–text retrieval

Image–Text Retrieval (ITR) aims to bridge the heterogeneity gap between image and text and establish the retrieval ability between the two modalities. There is a large intraclass difference in visual language data; that is, the content of an image can be described from different views. Representing images and texts as single embedded features to measure similarity makes it difficult to capture the diversity and fine-grained information of modal features. Previous methods use cumbersome dilated convolution structures or stack multiple feature pooling operators to perform multiview learning of images, and then take the view with the highest similarity score to the text as the alignment result. This may lead to two problems: (1) The model becomes complex, and has poor scalability and limited feature learning ability; (2) The matching strategy with the highest similarity score may have the problem that text view features deliberately emphasize a certain area, resulting in sub-optimal matching. Therefore, we propose a multiview adaptive attention pooling (MVAAP) network, a simpler and more effective multiview global feature embedding method. Specifically, MVAAP learns the query for each view, extracts the salient image region of the view through an adaptive attention mechanism, and generates an optimal pooling strategy to aggregate it into the global feature of the view. Beyond that, we introduce multiview embedding of the text branch and consider the response between different views to improve the generalization ability of the model. Plenty of experiments on the two mainstream cross-modal datasets of MS-COCO and Flickr30K prove the accuracy and superiority of the method.

Low-Rank Tensor Learning for Incomplete Multiview Clustering

Incomplete multiview clustering (IMVC) is an effective way to identify the underlying structure of incomplete multiview data. Most existing algorithms based on matrix factorization, graph learning or subspace learning have at least one of the following limitations: (1) the global and local structures of high-dimensional data are not effectively explored simultaneously; (2) the high-order correlations among multiple views are ignored. In this paper, we propose a low-rank tensor learning (LRTL) method that learns a consensus low-dimensional embedding matrix for IMVC. We first take advantage of the self-expressiveness property of high-dimensional data to construct sparse similarity matrices for individual views under low-rank and sparsity constraints. Individual low-dimensional embedding matrices can be obtained from the sparse similarity matrices using spectral embedding techniques. This approach simultaneously explores the global and local structures of incomplete multiview data. Then, we present a multiview embedding matrix fusion model that incorporates individual low-dimensional embedding matrices into a third-norm tensor to achieve a consensus low-dimensional embedding matrix. The fusion model exploits complementary information by finding the high-order correlations among multiple views. In addition, the computational cost of an improved fusion strategy is dramatically reduced. Extensive experimental results demonstrate that the proposed LRTL method outperforms several state-of-the-art approaches.

Deep multi-view graph clustering network with weighting mechanism and collaborative training

With the development of graph convolutional network (GCN), which is powerful in graph embedding learning meanwhile can capture node feature information, deep multi-view graph clustering methods based on graph autoencoder have emerged as a new stream. Although they achieve satisfactory performance, they still have the following weaknesses: (1) some of them model the weights for different views in the encoder part by using attention in the multi-view embedding fusion layer, but fail to consider the weighting in the decoder part to measure the contributions of different views to the reconstruction loss. (2) Most of them directly conduct clustering on the multi-view common embedding layer, but fail to guarantee the alignment of clustering results of different views. To this end, we propose a novel GCN-based deep multi-view graph clustering network with weighting mechanism and collaborative training (DMVGC). The model is composed of multiple view-specific graph encoders and a unified graph decoder. Besides the specially-designed attention module in the encoder part, we construct a reconstruction loss with adaptive weighting mechanism in the decoder part. Additionally, a collaborative self-training clustering objective is jointly conducted on each view-specific embedding layer and the common embedding layer, to make the embedding of each view clustering-friendly toward a common partition. Experiments on several datasets demonstrate the effectiveness of our model.

Zero-Shot Neural Decoding with Semi-Supervised Multi-View Embedding.

Zero-shot neural decoding aims to decode image categories, which were not previously trained, from functional magnetic resonance imaging (fMRI) activity evoked when a person views images. However, having insufficient training data due to the difficulty in collecting fMRI data causes poor generalization capability. Thus, models suffer from the projection domain shift problem when novel target categories are decoded. In this paper, we propose a zero-shot neural decoding approach with semi-supervised multi-view embedding. We introduce the semi-supervised approach that utilizes additional images related to the target categories without fMRI activity patterns. Furthermore, we project fMRI activity patterns into a multi-view embedding space, i.e., visual and semantic feature spaces of viewed images to effectively exploit the complementary information. We define several source and target groups whose image categories are very different and verify the zero-shot neural decoding performance. The experimental results demonstrate that the proposed approach rectifies the projection domain shift problem and outperforms existing methods.

Multiview Embedding with Partial Labels to Recognize Users of Devices Based on Unified Transformer

Recognizing the users of devices (or clusters of devices) who use IP addresses as unique identities on the Internet can easily enable numerous security applications. Fast and accurate user recognition is critical for supervisors to find influenced organizations connected to their networks in light of new security threats. Many users’ information scatters in the multisource data of IP addresses. Up until now, user recognition of devices has had two main problems. On the one hand, existing methods could not fully use multisource data of the IP addresses and wastes the valuable information of labels. On the other hand, only a tiny portion of devices can be tagged with highly confident known users manually, making it an urgent need to infer unknown users of devices. So, the problem of user recognition on devices is to guess the unknown user with multisource data and existing devices with known users. Therefore, this paper proposes a multiview fusion method to deal with multisource data from devices with a small number of manually labelled samples. The paper uses GraphSAGE to obtain an exemplary representation of IP addresses and designs a label encoder to fully use a small number of devices with known users. Then, the paper builds a specific unified transformer to achieve high performance to determine whether two devices have the same user. At the same time, the paper conducts real-world experiments and finds that the proposed method can achieve 0.9158 accuracy and 0.6131 F1 to find devices with the same users on the constructed dataset in the real world.

An intelligent recommendation method based on multi-interest network and adversarial deep learning

Recommender systems have shown to popular in many Internet communities, as they could help users discover interesting items based on their history behaviors. However, with the explosive growth of data-intensive tasks and online information, cybersecurity risks become larger, conventional collaborative recommendation algorithms may not meet users’ security requirements. Besides, the sparsity issue and the cold-start issue also hinder the performance of conventional recommendation methods. Recently, deep learning has shown to outperform traditional modeling techniques, which can be employed in Recommender systems (RSs) to improve user behavior prediction. In light of these challenges and observations, an intelligent recommendation method based on multi-interest network and adversarial deep learning is proposed, where multi-source behavior information is applied for multi-view embedding extraction for better prediction performance. Specifically, multi-view preference embeddings, including self-embedding, interaction-aware embedding, and neighbor-based embedding, are combined to model users’ interests at a finer granularity. Besides, in neighbor-based embedding learning, an adversarial search scheme is adopted for fast similarity searching and privacy preservation. Finally, a DNN-based prediction mechanism is adopted for embedding aggregation and final prediction. Extensive experiments on real-world datasets show that our proposal achieves decent prediction performance with security concerns compared with state-of the-art baselines.

Center consistency guided multi-view embedding anchor learning for large-scale graph clustering

Multi-view clustering has attracted extensive attention since it can integrate the complementary information of different views. Nonetheless, most existing methods suffer from cubic time computational complexity, so they have to face the challenge of maladjustment to medium and large-scale datasets. Whereupon, researchers usually employ anchor selection strategies to select a proportion of data points as landmarks to simulate the original data, and then build a bipartite graph to bridge the relationship between the original data points and anchors. However, for multi-view data, the existing strategies have three shortcomings, including: (1) on account of redundancy and corruption, the quality of anchors may be greatly affected; (2) these strategies are static ones or deficiency of views consistency; and (3) due to the differences in source, latent distribution, and the dimension of each view, the consensus bipartite graph cannot exploit the comprehensive information of all views precisely. In order to address these problems, this paper first proposes a novel method for scalable clustering, dubbed center consistency guided multi-view embedding anchor learning (2C-MEAL). Specifically, considering the redundancy and corruption, embedding learning is adopted to filter out the adverse information, as well as unify the dimensions of all views. Then, dynamic center consistency guided anchor learning is proposed to adaptively learn a set of high-quality anchors in the clean embedding space. Meanwhile, the refined bipartite graph is obtained for clustering purposes. Overall, 2C-MEAL has promising clustering performance and linear complexity. Experimentally, clustering experiments on benchmark datasets demonstrate that 2C-MEAL possesses effectiveness and efficiency over state-of-the-art methods.

DMvLNet: deep multiview learning network for blindly accessing image quality

In image quality assessment (IQA), the quality of an image is essentially dependent on both semantic contents and image distortions. Conventional nonreference IQA (NR-IQA) methods either encode the semantic contents only or combine them in a straightforward manner without exploring their intrinsic correlation. A unified deep multiview learning network (DMvLNet) is proposed to combine both semantic contents and image distortions for NR-IQA. Using ResNet50 as the backbone, we design IQA-aware semantic features instead of directly using a recognition-specific CNN model for IQA, as in many existing methods. To this end, a high-level feature refining module built on deep layers is developed for generating refined high-level feature maps. Meanwhile, shallow layers are employed for low-level descriptions. Thus, both the refined high-level and low-level features are used for characterizing semantic contents and image distortions, respectively. Furthermore, the refined semantic features and distortion features are simultaneously integrated into DMvLNet by exploring their mutual correlation, leading to the projected multiview embedding passing through the subsequent transformer network for image quality prediction. Extensive experiments conducted on both our assembled large-scale dataset and seven public benchmark datasets demonstrate the superiority of DMvLNet over the state-of-the-art methods.

Privacy-Aware Factorization-Based Hybrid Recommendation Method for Healthcare Services

With the advancements of the Health 2.0 technology, large-scale healthcare services are available online. Recommender systems for healthcare services have emerged for decision assistance. Most existing collaborative recommendation algorithms only mine global interactions while failing to capture the local different information of users or items. Besides, privacy concern is another significant problem to be considered in recommender systems for healthcare services. In this article, a privacy-aware factorization-based hybrid method is proposed for healthcare service recommendations. For better modeling of user preferences and service features, multiview embeddings of users and healthcare services are learned. Besides, we address the privacy problem by integrating local differential privacy and locality-sensitive hashing techniques into the recommendation model for privacy-aware neighbor searching. The final prediction is made based on a stochastic gradient descent learning-based hybrid collaborative model. Experiments demonstrate the effectiveness of the proposed method in both recommendation performance and privacy concerns.

Heterogeneous graph neural network for attribute completion

Heterogeneous graphs consist of multiple types of nodes and edges, and contain comprehensive information and rich semantics, which can properly model real-world complex systems. However, the attribute values of nodes are often incomplete with many missing attributes, as the cost of collecting node attributes is prohibitively expensive or even impossible (e.g., sensitive personal information). While a handful of graph neural network (GNN) models are developed for attribute completion in heterogeneous networks, most of them either ignore the use of similarity between nodes in feature space, or overlook the different importance of different-order neighbor nodes for attribute completion, resulting in poor performance. In this paper, we propose a general Attribute Completion framework for HEterogeneous Networks (AC-HEN), which is composed of feature aggregation, structure aggregation, and multi-view embedding fusion modules. Specifically, AC-HEN leverages feature aggregation and structure aggregation to obtain multi-view embeddings considering neighbor aggregation in both feature space and network structural space, which distinguishes different contributions of different neighbor nodes by conducting weighted aggregation. Then AC-HEN uses the multi-view embeddings to complete the missing attributes via an embedding fusion module in a weak supervised learning paradigm. Extensive experiments on three real-world heterogeneous network datasets demonstrate the superiority of AC-HEN against state-of-the-art baselines in both attribute completion and node classification. The source code is available at: https://github.com/Code-husky/AC-HEN.

A Survey of Entity Alignment of Knowledge Graph Based on Embedded Representation

This paper summarizes the main methods of knowledge representation learning. Representation learning represents the entity information of the knowledge graph as a low dimensional vector. Its vector dimension is low, which helps to improve the computational efficiency and make full use of the semantic information between entities. In order to embed two KGs into a unified semantic space, joint embedding is used to achieve this goal. With the development of research, there are many improved embedding methods, such as iteration, multi view embedding, knowledge graph entity alignment based on graph neural network and so on.

Experimental study on prediction of nonlinear system based on causality test

The prediction of nonlinear and non-stationary systems is a research topic of great scientific significance. In some recent work the convergent cross mapping (CCM) algorithm is used to detect the causal relationship between variables. In the CCM algorithm, the points close to each other in the phase space have similar trends and trajectories in time. Therefore, this method can be applied to the prediction of experimental researches of nonlinear and non-stationary systems. Therefore, in this paper the CCM algorithm is applied to the prediction of the Lorenz system and the actual climate time series, and the effects of different phase space reconstruction methods on the prediction skill are investigated. The preliminary results are as follows. 1) No matter whether the ideal Lorenz model or the actual climate series, of the three reconstruction phase space methods of univariate, multivariate, and multiview embedding method, the multiview embedding method is the best predictive skill, indicating that for a given length of time series, the more the information contained in the reconstructed phase space, the stronger its predictive ability is. 2) Adding the data of NAM (northern hemisphere annular mode) to the reconstructed phase space of SAT (surface air temperature) can improve the prediction effect on prediction of SAT. Using the univariable, multivariable, and multiview embedding method for implementing prediction, the characteristics of common information in the complex system are considered. On condition that the length of the time series is fixed, the complexity of the dynamic system can be used to increase the information of the system. Based on causality detection, through the extraction of quantitative information of data, a novel idea for the improvement of predictive skills in nonlinear and non-stationary systems can be obtained.

Medium-term prediction of key chemical process parameter trend with small data

Accidents might happen in chemical factories when less experienced operators unreasonably react to emergencies. It is therefore desirable to predict key process parameters early enough to help plant management easily and readily assess the coming risks and take proper actions to avoid them. Some of the predictions are in the time scale of days, if not months. However, such tools are rarely available in the chemical process industry, and often require a large amount of data for modeling. With inspiration from ecological researches, we propose a novel integrated causal analysis multiview embedding(ICAME) algorithm to predict medium-term trends of key chemical process parameters with small datasets. The proposed ICAME is mainly based on empirical dynamic modeling(EDM), a group of time series analysis approaches. Its application in a simulated dataset and a real industrial process demonstrated its advantages on small data over other methods for medium-term trend predictions of chemical process parameters.

Prediction of surface roughness of various machining processes by a hybrid algorithm including time series analysis, wavelet transform and multi view embedding

The momentary control of manufacturing processes is one of the ways to increase the productivity of production lines. The online measurement of the surface roughness by non-contact methods can be utilized in order to predict the future of surface texture and modify the machining parameters. In the technique proposed at this paper, the surface texture is extracted by combining the 2D surface photography and wavelet approach. Then, by extracting the time delay parameters, the embedding dimension and the false nearest neighbor of the produced surface texture, the future surface roughness is predicted. The results show that this technique can be used in lapping, grinding, turning, and milling processes. Although the maximum roughness error occurred in the surface roughness prediction is 24%, the prediction error is almost constant after Ra = 0.4 μm in different machining processes (about 7%). This study is in line with the development of the proposed method by Pour (2018).

Similarity-driven multi-view embeddings from high-dimensional biomedical data.

Diverse, high-dimensional modalities collected in large cohorts present new opportunities for the formulation and testing of integrative scientific hypotheses. Similarity-driven multi-view linear reconstruction (SiMLR) is an algorithm that exploits inter-modality relationships to transform large scientific datasets into smaller, more well-powered and interpretable low-dimensional spaces. SiMLR contributes an objective function for identifying joint signal, regularization based on sparse matrices representing prior within-modality relationships and an implementation that permits application to joint reduction of large data matrices. We demonstrate that SiMLR outperforms closely related methods on supervised learning problems in simulation data, a multi-omics cancer survival prediction dataset and multiple modality neuroimaging datasets. Taken together, this collection of results shows that SiMLR may be applied to joint signal estimation from disparate modalities and may yield practically useful results in a variety of application domains.

Generalized Multiview Shared Subspace Learning Using View Bootstrapping

A key objective in multi-view learning is to model the information common to multiple parallel views of a class of objects/events to improve downstream learning tasks. In this context, two open research questions remain: How can we model hundreds of views per event? Can we learn robust multi-view embeddings without any knowledge of how these views are acquired? We present a neural method based on multi-view correlation to capture the information shared across a large number of views by subsampling them in a view-agnostic manner during training. To provide an upper bound on the number of views to subsample for a given embedding dimension, we analyze the error of the bootstrapped multi-view correlation objective using matrix concentration theory. Our experiments on spoken word recognition, 3D object classification and pose-invariant face recognition demonstrate the robustness of view bootstrapping to model a large number of views. Results underscore the applicability of our method for a view-agnostic learning setting.