Multi-view Feature Research Articles

Multi-view Feature Learning Based on Texture Description for Palm-Print Recognition

Multi-view Feature Learning Based on Texture Description for Palm-Print Recognition

Self-supervised Multi-view Learning via Auto-encoding 3D Transformations

3D object representation learning is a fundamental challenge in computer vision to infer about the 3D world. Recent advances in deep learning have shown their efficiency in 3D object recognition, among which view-based methods have performed best so far. However, feature learning of multiple views in existing methods is mostly performed in a supervised fashion, which often requires a large amount of data labels with high costs. In contrast, self-supervised learning aims to learn multi-view feature representations without involving labeled data. To this end, we propose a novel self-supervised framework to learn Multi-View Transformation Equivariant Representations (MV-TER), exploring the equivariant transformations of a 3D object and its projected multiple views that we derive. Specifically, we perform a 3D transformation on a 3D object and obtain multiple views before and after the transformation via projection. Then, we train a representation encoding module to capture the intrinsic 3D object representation by decoding 3D transformation parameters from the fused feature representations of multiple views before and after the transformation. Experimental results demonstrate that the proposed MV-TER significantly outperforms the state-of-the-art view-based approaches in 3D object classification and retrieval tasks and show the generalization to real-world datasets. The code is available at https://github.com/gyshgx868/mvter .

Learning Multi-Types of Neighbor Node Attributes and Semantics by Heterogeneous Graph Transformer and Multi-View Attention for Drug-Related Side-Effect Prediction.

Since side-effects of drugs are one of the primary reasons for their failure in clinical trials, predicting their side-effects can help reduce drug development costs. We proposed a method based on heterogeneous graph transformer and capsule networks for side-effect-drug-association prediction (TCSD). The method encodes and integrates attributes from multiple types of neighbor nodes, connection semantics, and multi-view pairwise information. In each drug-side-effect heterogeneous graph, a target node has two types of neighbor nodes, the drug nodes and the side-effect ones. We proposed a new heterogeneous graph transformer-based context representation learning module. The module is able to encode specific topology and the contextual relations among multiple kinds of nodes. There are similarity and association connections between the target node and its various types of neighbor nodes, and these connections imply semantic diversity. Therefore, we designed a new strategy to measure the importance of a neighboring node to the target node and incorporate different semantics of the connections between the target node and its multi-type neighbors. Furthermore, we designed attentions at the neighbor node type level and at the graph level, respectively, to obtain enhanced informative neighbor node features and multi-graph features. Finally, a pairwise multi-view feature learning module based on capsule networks was built to learn the pairwise attributes from the heterogeneous graphs. Our prediction model was evaluated using a public dataset, and the cross-validation results showed it achieved superior performance to several state-of-the-art methods. Ablation experiments undertaken demonstrated the effectiveness of heterogeneous graph transformer-based context encoding, the position enhanced pairwise attribute learning, and the neighborhood node category-level attention. Case studies on five drugs further showed TCSD's ability in retrieving potential drug-related side-effect candidates, and TCSD inferred the candidate side-effects for 708 drugs.

Multi-View Feature Aggregation for Predicting Microbe-Disease Association.

Microbes play a crucial role in human health and disease. Figuring out the relationship between microbes and diseases leads to significant potential applications in disease treatments. It is an urgent need to devise robust and effective computational methods for identifying disease-related microbes. This work proposes a Multi-View Feature Aggregation (MVFA) scheme that integrates the linear and nonlinear features to identify disease-related microbes. We introduce a non-negative matrix tri-factorization (NMTF) model to extract linear features for diseases and microbes. Then we learn another type of linear feature by utilizing a bi-random walk model. The nonlinear feature is obtained by inputting the two kinds of linear features into a capsule neural network. These three types of features describe the associations between diseases and microbes from different views. Finally, considering the complementary of these features, we leverage a logistic regression model to combine the NMTF model predictions, bi-random walk model predictions, and the capsule neural network predictions to obtain the final microbe-disease pair scores. We apply our method to predict human microbe-disease associations on two datasets. Experimental results show that our multi-view model outperforms the state-of-the-art models in recovering missing microbe-disease associations and predicting associations for new microbes. The ablation study shows that aggregating multi-view linear and nonlinear features can improve the prediction performance. Case studies on two diseases, i.e. Type 1 diabetes and Liver cirrhosis, further validate our method effectiveness.

URRNet: A Unified Relational Reasoning Network for Vehicle Re-Identification

With the continuous improvement and optimization of security monitoring networks, vehicle Re-Identification (Re-ID) becomes an emerging key technology in the development of intelligent visual surveillance systems. Due to the influence of viewpoint variation and fine-grained differences, vehicle Re-ID is still a research topic worth investigating. To alleviate above problems, a novel end-to-end framework named Unified Relational Reasoning Network (URRNet) is proposed in this paper, which integrates global features with local features to obtain better recognition accuracy. For the proposed framework, to understand the overall semantics of the image, an algorithm based on the global feature graph-structure learning is designed. The pixel-level feature maps are transformed to the node features of graph in the interactive space by projection, then graph reasoning is performed by using the graph convolutional network to improve the representation of global features. Moreover, an algorithm based on multi-scale local feature relational reasoning is designed. Using keypoint and viewpoint to obtain the multi-scale partial characteristics of the vehicle, and the vehicle multi-view features are learned from the single-view vehicle images through relational reasoning and attention mechanism. The two algorithms are combined to obtain the overall model, which not only preserves the details of the vehicle, but also effectively solves the problem of viewpoint variation. Comprehensive experimental results on two public datasets (VeRi-776 and VehicleID) indicate that the proposed URRNet can practically improve the model's representation ability and generalization ability, which is comparable to the state-of-the-art vehicle Re-ID methods.

SenticGAT: Sentiment Knowledge Enhanced Graph Attention Network for Multi-view Feature Representation in Aspect-based Sentiment Analysis

Currently, computational intelligence methods, especially artificial neural networks, are increasingly applied to many scenarios. We mainly attempt to explore the task of fine-grained sentiment classification of review data through computational intelligence methods, especially artificial neural networks, and this task is also known as aspect-based sentiment analysis (ABSA). We propose a new technique called SenticGAT which is a multi-view features fusion model enhanced by an external sentiment database. We encode the external sentiment information into the syntactic dependency tree to obtain an enhanced graph with rich sentiment representation. Then we obtain multi-view features including semantics, syntactic, and sentiment features through GAT based on the enhanced graph by external knowledge. We also design a new strategy for fusing multi-view features using the feature parallel frame and convolution method. Eventually, the sentiment polarity of a specific aspect is determined based on the completely fused multi-view features. Experimental results on four public benchmark datasets demonstrate that our method is effective and sound. And it performs superiorly to existing approaches in fusion multiple-view features.

Local climate zone mapping using remote sensing: a synergetic use of daytime multi-view Ziyuan-3 stereo imageries and Luojia-1 nighttime light data

ABSTRACT The local climate zone (LCZ) scheme has been widely utilized in regional climate modeling, urban planning, and thermal comfort investigations. However, existing LCZ classification methods face challenges in characterizing complex urban structures and human activities involving local climatic environments. In this study, we proposed a novel LCZ mapping method that fully uses space-borne multi-view and diurnal observations, i.e. daytime Ziyuan-3 stereo imageries (2.1 m) and Luojia-1 nighttime light (NTL) data (130 m). Firstly, we performed land cover classification using multiple machine learning methods (i.e. random forest (RF) and XGBoost algorithms) and various features (i.e. spectral, textural, multi-view features, 3D urban structure parameters (USPs), and NTL). In addition, we developed a set of new cumulative elevation indexes to improve building roughness assessments. The indexes can estimate building roughness directly from fused point clouds generated by both along- and across-track modes. Finally, based on the land cover and building roughness results, we extracted 2D and 3D USPs for different land covers and used multi-classifiers to perform LCZ mapping. The results for Beijing, China, show that our method yielded satisfactory accuracy for LCZ mapping, with an overall accuracy (OA) of 90.46%. The overall accuracy of land cover classification using 3D USPs generated from both along- and across-track modes increased by 4.66%, compared to that of using the single along-track mode. Additionally, the OA value of LCZ mapping using 2D and 3D USPs (88.18%) achieved a better result than using only 2D USPs (83.83%). The use of NTL data increased the classification accuracy of LCZs E (bare rock or paved) and F (bare soil or sand) by 6.54% and 3.94%, respectively. The refined LCZ classification achieved through this study will not only contribute to more accurate regional climate modeling but also provide valuable guidance for urban planning initiatives aimed at enhancing thermal comfort and overall livability in urban areas. Ultimately, this study paves the way for more comprehensive and effective strategies in addressing the challenges posed by urban microclimates.

Multi-view enhanced zero-shot node classification

In recent years, Zero-shot Node Classification (ZNC), an emerging and more difficult task is starting to attract attention, where the classes of testing nodes are unobserved in the training stage. Existing studies for ZNC mainly utilize Graph Neural Networks (GNNs) to construct the feature subspace to align with the classes’ semantic subspace, thus enabling knowledge transfer from seen classes to unseen classes. However, the modeling of the node feature is single-view and unilateral, e.g., the bag-of-words vector, which is not enough to fully describe the characteristics of the node itself. To address this dilemma, we propose to develop the Multi-View Enhanced zero-shot node classification paradigm (MVE) to promote the machine’s generality to approach the human-like thinking mode. Specifically, multi-view features are obtained from different aspects such as pre-trained model embeddings, knowledge graphs, statistic methods, and then fused by a contrastive learning module into the compositional node representation. Meanwhile, a developed Graph Convolutional Network (GCN) is used to make the nodes fully absorb the information of neighbors while the over-smooth issue is alleviated by multi-view features and the proposed contrastive learning mechanism. Experimental results conducted on three public datasets show an average 25% improvement compared to baseline methods, proving the superiority of our multi-view learning framework. The code and data can be found at https://github.com/guaiqihen/MVE.

Robust image hashing based on multi-view dimension reduction

In the field of multimedia security, image hashing is an effective technology for solving image content identification. Image hashing algorithm aims to map the high dimensional input features onto perceptual hashing values in terms of their perceptual content, in which visually identical images are mapped to similar digital representations. In this paper, we focus on multi-view dimensionality reduction based hashing generation, and propose a novel method, called Multi-view Dimensionality Reduction based Robust Image Hashing (MDRIH). Our MDRIH scheme maps the multi-view features into a compact binary space by considering the complementarity and similarity between image features across these different views. For preprocessing, bilinear interpolation is applied on the host image to adjust to a standard size, and ensure the different images with same hash size. Specifically, in order to improve the robustness of the algorithm against content-preserving operations, multi-view features, including structural features, edge features, and color features are extracted. After that, multi-view dimension reduction is adopted to obtain meaningful low-dimensional information from multi-feature fusion. Finally, the data in low-dimensional subspace is encoded into a compact binary sequence. The experimental results indicate that the proposed hashing method not only preserves the correlation between multiple views and the similarity among different data points within each view, but also achieves a better balance between robustness and discriminability compared to existing hashing methods.

HAR_Locator: a novel protein subcellular location prediction model of immunohistochemistry images based on hybrid attention modules and residual units.

Introduction: Proteins located in subcellular compartments have played an indispensable role in the physiological function of eukaryotic organisms. The pattern of protein subcellular localization is conducive to understanding the mechanism and function of proteins, contributing to investigating pathological changes of cells, and providing technical support for targeted drug research on human diseases. Automated systems based on featurization or representation learning and classifier design have attracted interest in predicting the subcellular location of proteins due to a considerable rise in proteins. However, large-scale, fine-grained protein microscopic images are prone to trapping and losing feature information in the general deep learning models, and the shallow features derived from statistical methods have weak supervision abilities. Methods: In this work, a novel model called HAR_Locator was developed to predict the subcellular location of proteins by concatenating multi-view abstract features and shallow features, whose advanced advantages are summarized in the following three protocols. Firstly, to get discriminative abstract feature information on protein subcellular location, an abstract feature extractor called HARnet based on Hybrid Attention modules and Residual units was proposed to relieve gradient dispersion and focus on protein-target regions. Secondly, it not only improves the supervision ability of image information but also enhances the generalization ability of the HAR_Locator through concatenating abstract features and shallow features. Finally, a multi-category multi-classifier decision system based on an Artificial Neural Network (ANN) was introduced to obtain the final output results of samples by fitting the most representative result from five subset predictors. Results: To evaluate the model, a collection of 6,778 immunohistochemistry (IHC) images from the Human Protein Atlas (HPA) database was used to present experimental results, and the accuracy, precision, and recall evaluation indicators were significantly increased to 84.73%, 84.77%, and 84.70%, respectively, compared with baseline predictors.

Multi-View Graph Convolutional Networks with Differentiable Node Selection

Multi-view data containing complementary and consensus information can facilitate representation learning by exploiting the intact integration of multi-view features. Because most objects in the real world often have underlying connections, organizing multi-view data as heterogeneous graphs is beneficial to extracting latent information among different objects. Due to the powerful capability to gather information of neighborhood nodes, in this article, we apply Graph Convolutional Network (GCN) to cope with heterogeneous graph data originating from multi-view data, which is still under-explored in the field of GCN. In order to improve the quality of network topology and alleviate the interference of noises yielded by graph fusion, some methods undertake sorting operations before the graph convolution procedure. These GCN-based methods generally sort and select the most confident neighborhood nodes for each vertex, such as picking the top- k nodes according to pre-defined confidence values. Nonetheless, this is problematic due to the non-differentiable sorting operators and inflexible graph embedding learning, which may result in blocked gradient computations and undesired performance. To cope with these issues, we propose a joint framework dubbed Multi-view Graph Convolutional Network with Differentiable Node Selection (MGCN-DNS), which is constituted of an adaptive graph fusion layer, a graph learning module, and a differentiable node selection schema. MGCN-DNS accepts multi-channel graph-structural data as inputs and aims to learn more robust graph fusion through a differentiable neural network. The effectiveness of the proposed method is verified by rigorous comparisons with considerable state-of-the-art approaches in terms of multi-view semi-supervised classification tasks, and the experimental results indicate that MGCN-DNS achieves pleasurable performance on several benchmark multi-view datasets.

Double High-Order Correlation Preserved Robust Multi-View Ensemble Clustering

Ensemble clustering (EC), utilizing multiple basic partitions (BPs) to yield a robust consensus clustering, has shown promising clustering performance. Nevertheless, most current algorithms suffer from two challenging hurdles: (1) a surge of EC-based methods only focus on pair-wise sample correlation while fully ignoring the high-order correlations of diverse views. (2) they deal directly with the co-association (CA) matrices generated from BPs, which are inevitably corrupted by noise and thus degrade the clustering performance. To address these issues, we propose a novel Double High-Order Correlation Preserved Robust Multi-View Ensemble Clustering (DC-RMEC) method, which preserves the high-order inter-view correlation and the high-order correlation of original data simultaneously. Specifically, DC-RMEC constructs a hypergraph from BPs to fuse high-level complementary information from different algorithms and incorporates multiple CA-based representations into a low-rank tensor to discover the high-order relevance underlying CA matrices, such that double high-order correlation of multi-view features could be dexterously uncovered. Moreover, a marginalized denoiser is invoked to gain robust view-specific CA matrices. Furthermore, we develop a unified framework to jointly optimize the representation tensor and the result matrix. An effective iterative optimization algorithm is designed to optimize our DC-RMEC model by resorting to the alternating direction method of multipliers. Extensive experiments on seven real-world multi-view datasets have demonstrated the superiority of DC-RMEC compared with several state-of-the-art multi-view ensemble clustering methods.

Multiview Feature Selection With Information Complementarity and Consensus for Fault Diagnosis

Feature selection plays a critical role in data-driven fault diagnosis. Existing works often perform feature selection on a single overcomplete feature pool that is collected from various feature extraction approaches, such as signal processing and deep learning. However, simple concatenation of these “multiview” features may neglect the inherent properties and cross-feature correlations, often leading to ineffectiveness in feature selection for fault diagnosis. Therefore, a multiview feature selection with information complementarity and consensus (MFSICC) is proposed in this article to efficiently select relevant features for fault diagnosis. In particular, both complementary and consensus properties that are shared among the multiview features are incorporated based on a structured sparsity learning model. Besides, an iterative algorithm is proposed to solve the MFSICC problem with theoretically approved convergence. Experiments on bearing and gearbox fault diagnosis with real-world datasets validate the effectiveness of MFSICC. Important model parameters and empirical convergence are also discussed to facilitate practical use.

IoT-Enabled Few-Shot Image Generation for Power Scene Defect Detection Based on Self-Attention and Global-Local Fusion.

Defect detection in power scenarios is a critical task that plays a significant role in ensuring the safety, reliability, and efficiency of power systems. The existing technology requires enhancement in its learning ability from large volumes of data to achieve ideal detection effect results. Power scene data involve privacy and security issues, and there is an imbalance in the number of samples across different defect categories, all of which will affect the performance of defect detection models. With the emergence of the Internet of Things (IoT), the integration of IoT with machine learning offers a new direction for defect detection in power equipment. Meanwhile, a generative adversarial network based on multi-view fusion and self-attention is proposed for few-shot image generation, named MVSA-GAN. The IoT devices capture real-time data from the power scene, which are then used to train the MVSA-GAN model, enabling it to generate realistic and diverse defect data. The designed self-attention encoder focuses on the relevant features of different parts of the image to capture the contextual information of the input image and improve the authenticity and coherence of the image. A multi-view feature fusion module is proposed to capture the complex structure and texture of the power scene through the selective fusion of global and local features, and improve the authenticity and diversity of generated images. Experiments show that the few-shot image generation method proposed in this paper can generate real and diverse defect data for power scene defects. The proposed method achieved FID and LPIPS scores of 67.87 and 0.179, surpassing SOTA methods, such as FIGR and DAWSON.

Adaptive Collaborative Soft Label Learning for Unsupervised Multi-View Feature Selection

Unsupervised multi-view feature selection aims to select informative features with multi-view features and unsupervised learning. It is a challenging problem due to the absence of explicit semantic supervision. Recently, graph theory and hard pseudo-label learning have been adopted to solve multi-view feature selection problems under the unsupervised learning paradigm. However, graph-based methods are difficult to support large-scale real scenarios due to the high computational complexity of graph construction. Moreover, existing methods based on hard pseudo-label learning generally result in significant information loss. In this article, we propose an Adaptive Collaborative Soft Label Learning (ACSLL) model for unsupervised multi-view feature selection. In this model, collaborative soft label learning and multi-view feature selection are integrated into a unified framework. Specifically, we learn the pseudo soft labels from each view feature by a simple and efficient method and fuse them with an adaptive weighting strategy into a joint soft label matrix. This matrix is further used for guiding the feature selection process to identify valuable features. An effective optimization strategy guaranteed with proven convergence is derived to iteratively solve this problem. Experiments demonstrate the superiority of the proposed method in both feature selection accuracy and efficiency.

DICNet: Deep Instance-Level Contrastive Network for Double Incomplete Multi-View Multi-Label Classification

In recent years, multi-view multi-label learning has aroused extensive research enthusiasm. However, multi-view multi-label data in the real world is commonly incomplete due to the uncertain factors of data collection and manual annotation, which means that not only multi-view features are often missing, and label completeness is also difficult to be satisfied. To deal with the double incomplete multi-view multi-label classification problem, we propose a deep instance-level contrastive network, namely DICNet. Different from conventional methods, our DICNet focuses on leveraging deep neural network to exploit the high-level semantic representations of samples rather than shallow-level features. First, we utilize the stacked autoencoders to build an end-to-end multi-view feature extraction framework to learn the view-specific representations of samples. Furthermore, in order to improve the consensus representation ability, we introduce an incomplete instance-level contrastive learning scheme to guide the encoders to better extract the consensus information of multiple views and use a multi-view weighted fusion module to enhance the discrimination of semantic features. Overall, our DICNet is adept in capturing consistent discriminative representations of multi-view multi-label data and avoiding the negative effects of missing views and missing labels. Extensive experiments performed on five datasets validate that our method outperforms other state-of-the-art methods.

Delving Deep into Pixel Alignment Feature for Accurate Multi-View Human Mesh Recovery

Regression-based methods have shown high efficiency and effectiveness for multi-view human mesh recovery. The key components of a typical regressor lie in the feature extraction of input views and the fusion of multi-view features. In this paper, we present Pixel-aligned Feedback Fusion (PaFF) for accurate yet efficient human mesh recovery from multi-view images. PaFF is an iterative regression framework that performs feature extraction and fusion alternately. At each iteration, PaFF extracts pixel-aligned feedback features from each input view according to the reprojection of the current estimation and fuses them together with respect to each vertex of the downsampled mesh. In this way, our regressor can not only perceive the misalignment status of each view from the feedback features but also correct the mesh parameters more effectively based on the feature fusion on mesh vertices. Additionally, our regressor disentangles the global orientation and translation of the body mesh from the estimation of mesh parameters such that the camera parameters of input views can be better utilized in the regression process. The efficacy of our method is validated in the Human3.6M dataset via comprehensive ablation experiments, where PaFF achieves 33.02 MPJPE and brings significant improvements over the previous best solutions by more than 29%. The project page with code and video results can be found at https://kairobo.github.io/PaFF/.

MVIL6: Accurate identification of IL-6-induced peptides using multi-view feature learning

Interleukin-6 (IL-6) is a potential therapeutic target for many diseases, and it is of great significance in accurately predicting IL-6-induced peptides for IL-6 research. However, the cost of traditional wet experiments to detect IL-6-induced peptides is huge, and the discovery and design of peptides by computer before the experimental stage have become a promising technology. In this study, we developed a deep learning model called MVIL6 for predicting IL-6-inducing peptides. Comparative results demonstrated the outstanding performance and robustness of MVIL6. Specifically, we employ a pre-trained protein language model MG-BERT and the Transformer model to process two different sequence-based descriptors and integrate them with a fusion module to improve the prediction performance. The ablation experiment demonstrated the effectiveness of our fusion strategy for the two models. In addition, to provide good interpretability of our model, we explored and visualized the amino acids considered important for IL-6-induced peptide prediction by our model. Finally, a case study presented using MVIL6 to predict IL-6-induced peptides in the SARS-CoV-2 spike protein shows that MVIL6 achieves higher performance than existing methods and can be useful for identifying potential IL-6-induced peptides in viral proteins.

Variational gated autoencoder-based feature extraction model for inferring disease-miRNA associations based on multiview features

MicroRNAs (miRNA) play critical roles in diverse biological processes of diseases. Inferring potential disease-miRNA associations enable us to better understand the development and diagnosis of complex human diseases via computational algorithms. The work presents a variational gated autoencoder-based feature extraction model to extract complex contextual features for inferring potential disease-miRNA associations. Specifically, our model fuses three different similarities of miRNAs into a comprehensive miRNA network and then combines two various similarities of diseases into a comprehensive disease network, respectively. Then, a novel graph autoencoder is designed to extract multilevel representations based on variational gate mechanisms from heterogeneous networks of miRNAs and diseases. Finally, a gate-based association predictor is devised to combine multiscale representations of miRNAs and diseases via a novel contrastive cross-entropy function, and then infer disease-miRNA associations. Experimental results indicate that our proposed model achieves remarkable association prediction performance, proving the efficacy of the variational gate mechanism and contrastive cross-entropy loss for inferring disease-miRNA associations.

Graph based encrypted malicious traffic detection with hybrid analysis of multi-view features

At present, the TLS cryptographic protocol is widely deployed. While protecting the security and integrity of transmitted information, it also makes the detection of malicious behavior more difficult. In recent years, researchers have proposed many encrypted malicious traffic detection methods. However, the existing approaches have some shortcomings. Firstly, although researchers have extracted multi-view features from different aspects, which can be divided into vectorized features based on feature engineering and image features based on original data, existing methods cannot fully integrate the features of different forms of expression. Secondly, most of the existing methods do not fully analyze the correlation between different encrypted traffic. Thirdly, the existing methods based on correlation analysis have low processing efficiency and cannot be applied to real networks. In the paper, we present MalDiscovery, a novel technique to discover encrypted malicious traffic to address all the above issues. For encrypted malicious traffic, MalDiscovery constructs an attribute KNN graph, in which encrypted sessions are used as nodes to construct a KNN graph according to the similarity of image features, and vectorized features are used as attributes of corresponding nodes. After that, the GraphSAGE model is used to collect relevant node information through correlation analysis to enrich the embeddings of each node. Finally, we achieve the accurate binary classification of nodes in the graph based on richer embeddings. We use extensive experiments to evaluate the proposed method, and the experiment results show that MalDiscovery can achieve an accuracy of about 99.9%, significantly outperforming all compared methods.