Cross-view Correlation Research Articles

STViT+: improving self-supervised multi-camera depth estimation with spatial-temporal context and adversarial geometry regularization

Multi-camera depth estimation has gained significant attention in autonomous driving due to its importance in perceiving complex environments. However, extending monocular self-supervised methods to multi-camera setups introduces unique challenges that existing techniques often fail to address. In this paper, we propose STViT+, a novel Transformer-based framework for self-supervised multi-camera depth estimation. Our key contributions include: 1) the Spatial-Temporal Transformer (STTrans), which integrates local spatial connectivity and global context to capture enriched spatial-temporal cross-view correlations, resulting in more accurate 3D geometry reconstruction; 2) the Spatial-Temporal Photometric Consistency Correction (STPCC) strategy that mitigates the impact of varying illumination, ensuring brightness consistency across frames during photometric loss calculation; 3) the Adversarial Geometry Regularization (AGR) module, which employs Generative Adversarial Networks to impose spatial constraints by using unpaired depth maps, enhancing performance under adverse conditions such as rain and nighttime driving. Extensive evaluations on large-scale autonomous driving datasets, including Nuscenes and DDAD, confirm that STViT+ sets a new benchmark for multi-camera depth estimation.

Multi-View Integrative Approach For Imputing Short-Chain Fatty Acids and Identifying Key factors predicting Blood SCFA.

Short-chain fatty acids (SCFAs) are the main metabolites produced by bacterial fermentation of dietary fiber within gastrointestinal tract. SCFAs produced by gut microbiotas (GMs) are absorbed by host, reach bloodstream, and are distributed to different organs, thus influencing host physiology. However, due to the limited budget or the poor sensitivity of instruments, most studies on GMs have incomplete blood SCFA data, limiting our understanding of the metabolic processes within the host. To address this gap, we developed an innovative multi-task multi-view integrative approach (M2AE, Multi-task Multi-View Attentive Encoders), to impute blood SCFA levels using gut metagenomic sequencing (MGS) data, while taking into account the intricate interplay among the gut microbiome, dietary features, and host characteristics, as well as the nuanced nature of SCFA dynamics within the body. Here, each view represents a distinct type of data input (i.e., gut microbiome compositions, dietary features, or host characteristics). Our method jointly explores both view-specific representations and cross-view correlations for effective predictions of SCFAs. We applied M2AE to two in-house datasets, which both include MGS and blood SCFAs profiles, host characteristics, and dietary features from 964 subjects and 171 subjects, respectively. Results from both of two datasets demonstrated that M2AE outperforms traditional regression-based and neural-network based approaches in imputing blood SCFAs. Furthermore, a series of gut bacterial species (e.g., Bacteroides thetaiotaomicron and Clostridium asparagiforme), host characteristics (e.g., race, gender), as well as dietary features (e.g., intake of fruits, pickles) were shown to contribute greatly to imputation of blood SCFAs. These findings demonstrated that GMs, dietary features and host characteristics might contribute to the complex biological processes involved in blood SCFA productions. These might pave the way for a deeper and more nuanced comprehension of how these factors impact human health.

IRLF-SRNet: A super-resolution network based on local–global feature enhance-refine for camera-array based infrared light field images

Super resolution (SR) is essential for the camera-array based infrared light field (IRLF) images. However, the existing SR methods are mainly designed for visible LF images and are unsuitable for IRLF. This is because the sub-aperture images (SAIs) in IRLF hold blurred edges and pixel-level shifts. Artifacts and unbalanced SR performance may exist. In this paper, we propose a novel SR network, named IRLF-SRnet, to mitigate the above problems. It includes two designs: (1) local–global feature enhance-refine strategy, and (2) multi-branch cross-view correlation modeling block. The former uses global features from all SAIs to enhance the local feature from one SAI, which enhances the edges in each SAI implicitly. Then, it refines the global feature with the enhanced local feature, which further refines the edges. The latter first divides and sends the features of SAIs into multiple branches, and then models the cross-view correlation among features in each branch by bidirectional convolutional long-short term memory (BiConvLSTM), so that be aware of the pixel-level shifts. Qualitative and quantitative results show that our method outperforms state-of-the-art methods, with more balanced SR performance and fewer artifacts.

Incomplete Data Meets Uncoupled Case: A Challenging Task of Multiview Clustering.

Incomplete multiview clustering (IMC) methods have achieved remarkable progress by exploring the complementary information and consensus representation of incomplete multiview data. However, to our best knowledge, none of the existing methods attempts to handle the uncoupled and incomplete data simultaneously, which affects their generalization ability in real-world scenarios. For uncoupled incomplete data, the unclear and partial cross-view correlation introduces the difficulty to explore the complementary information between views, which results in the unpromising clustering performance for the existing multiview clustering methods. Besides, the presence of hyperparameters limits their applications. To fill these gaps, a novel uncoupled IMC (UIMC) method is proposed in this article. Specifically, UIMC develops a joint framework for feature inferring and recoupling. The high-order correlations of all views are explored by performing a tensor singular value decomposition (t-SVD)-based tensor nuclear norm (TNN) on recoupled and inferred self-representation matrices. Moreover, all hyperparameters of the UIMC method are updated in an exploratory manner. Extensive experiments on six widely used real-world datasets have confirmed the superiority of the proposed method in handling the uncoupled incomplete multiview data compared with the state-of-the-art methods.

STViT: Improving Self-Supervised Multi-Camera Depth Estimation with Spatial-Temporal Context and Adversarial Geometry Regularization (Student Abstract)

Multi-camera depth estimation has recently garnered significant attention due to its substantial practical implications in the realm of autonomous driving. In this paper, we delve into the task of self-supervised multi-camera depth estimation and propose an innovative framework, STViT, featuring several noteworthy enhancements: 1) we propose a Spatial-Temporal Transformer to comprehensively exploit both local connectivity and the global context of image features, meanwhile learning enriched spatial-temporal cross-view correlations to recover 3D geometry. 2) to alleviate the severe effect of adverse conditions, e.g., rainy weather and nighttime driving, we introduce a GAN-based Adversarial Geometry Regularization Module (AGR) to further constrain the depth estimation with unpaired normal-condition depth maps and prevent the model from being incorrectly trained. Experiments on challenging autonomous driving datasets Nuscenes and DDAD show that our method achieves state-of-the-art performance.

A Tensor Approach for Uncoupled Multiview Clustering.

Multiview clustering plays an important part in unsupervised learning. Although the existing methods have shown promising clustering performances, most of them assume that the data is completely coupled between different views, which is unfortunately not always ensured in real-world applications. The clustering performance of these methods drops dramatically when handling the uncoupled data. The main reason is that: 1) cross-view correlation of uncoupled data is unclear, which limits the existing multiview clustering methods to explore the complementary information between views and 2) features from different views are uncoupled with each other, which may mislead the multiview clustering methods to partition data into wrong clusters. To address these limitations, we propose a tensor approach for uncoupled multiview clustering (T-UMC) in this article. Instead of pairwise correlation, T-UMC chooses a most reliable view by view-specific silhouette coefficient (VSSC) at first, and then couples the self-representation matrix of each view with it by pairwise cross-view coupling learning. After that, by integrating recoupled self-representation matrices into a third-order tensor, the high-order correlations of all views are explored with tensor singular value decomposition (t-SVD)-based tensor nuclear norm (TNN). And the view-specific local structures of each individual view are also preserved with the local structure learning scheme with manifold learning. Besides, the physical meaning of view-specific coupling matrix is also discussed in this article. Extensive experiments on six commonly used benchmark datasets have demonstrated the superiority of the proposed method compared with the state-of-the-art multiview clustering methods.

Modeling Self-Representation Label Correlations for Textual Aspects and Emojis Recommendation.

The rapid development of Internet services and social platforms encourages users to share their opinions. To help users give valuable comments, content providers expect the recommender system to offer appropriate suggestions, including specific features of the item described in texts and emojis, which are all considered aspects of the user reviews. Hence, the review aspect recommendation task has become significant, where the key lies in handling personal preferences and semantic correlations between suggested items. This article proposes a correlation-aware review aspect recommender (CARAR) system model by constructing self-representation correlations between different views of review aspects, including textual aspects and emojis to make a personalized recommendation. The dependencies between different textual aspects and emojis can be identified and utilized to facilitate the factorization process to learn user and item latent factors. The cross-view correlation mapping between textual aspects and emojis can be built to enhance the recommendation performance. Moreover, the additional information in the real-world environment is also applied to our model to adjust the recommendation results. We constructed experiments on five self-collected and public datasets and compared with six existing models. The results show that our model can outperform the existing models on review aspects recommendation tasks, validating the effectiveness of our approach.

Weakly-Supervised 3D Human Pose Estimation With Cross-View U-Shaped Graph Convolutional Network

Although monocular 3D human pose estimation methods have made significant progress, it's far from being solved due to the inherent depth ambiguity. Instead, exploiting multi-view information is a practical way to achieve absolute 3D human pose estimation. In this paper, we propose a simple yet effective pipeline for weakly-supervised cross-view 3D human pose estimation. By only using two camera views, our method can achieve state-of-the-art performance in a weakly-supervised manner, requiring no 3D ground truth but only 2D annotations. Specifically, our method contains two steps: triangulation and refinement. First, given the 2D keypoints that can be obtained through any classic 2D detection methods, triangulation is performed across two views to lift the 2D keypoints into coarse 3D poses. Then, a novel cross-view U-shaped graph convolutional network (CV-UGCN), which can explore the spatial configurations and cross-view correlations, is designed to refine the coarse 3D poses. In particular, the refinement progress is achieved through weakly-supervised learning, in which geometric and structure-aware consistency checks are performed. We evaluate our method on the standard benchmark dataset, Human3.6M. The Mean Per Joint Position Error on the benchmark dataset is 27.4 mm, which outperforms the state-of-the-arts remarkably (27.4 mm vs 30.2 mm).

HEVC Based Frame Interleaved Coding Technique for Stereo and Multi-View Videos

The standard HEVC codec and its extension for coding multiview videos, known as MV-HEVC, have proven to deliver improved visual quality compared to its predecessor, H.264/MPEG-4 AVC’s multiview extension, H.264-MVC, for the same frame resolution with up to 50% bitrate savings. MV-HEVC’s framework is similar to that of H.264-MVC, which uses a multi-layer coding approach. Hence, MV-HEVC would require all frames from other reference layers decoded prior to decoding a new layer. Thus, the multi-layer coding architecture would be a bottleneck when it comes to quicker frame streaming across different views. In this paper, an HEVC-based Frame Interleaved Stereo/Multiview Video Codec (HEVC-FISMVC) that uses a single layer encoding approach to encode stereo and multiview video sequences is presented. The frames of stereo or multiview video sequences are interleaved in such a way that encoding the resulting monoscopic video stream would maximize the exploitation of temporal, inter-view, and cross-view correlations and thus improving the overall coding efficiency. The coding performance of the proposed HEVC-FISMVC codec is assessed and compared with that of the standard MV-HEVC’s performance for three standard multi-view video sequences, namely: “Poznan_Street”, “Kendo” and “Newspaper1”. Experimental results show that the proposed codec provides more substantial coding gains than the anchor MV-HEVC for coding both stereo and multi-view video sequences.

Intelligent Optimization Algorithm of 3D Tracking Technology in Football Player Moving Image Analysis

Computer vision technology began to affect the development of football. There is increasingly high-tech in football broadcast technology, and many application tools have emerged in the field of football broadcast video analysis. The purpose of this paper is to study the improvement of target tracking algorithm for football broadcast video and to study the intelligent optimization algorithm of 3D tracking technology in football player moving image analysis. This paper proposes to select four models of YOLOv5 to perform target detection experiments in football broadcast videos and analyzes the principle of the Deep SORT multitarget tracking algorithm. At the same time, it is based on the 3D tracking and 3D pose estimation of players based on cross-view correlation matching, and to measure the comprehensive performance of the tracker in the football scene, experiments are carried out on the accuracy and speed of the tracker under the football datasets of four different scenes. The experimental results in this paper show that the MOTA values corresponding to the 3D tracking results and 2D projection results obtained in the campus dataset are only 50 and 56.2. This is much lower than the tracking performance when based on other similarity matrices. The MOTA value of the obtained tracking result (92.6) is very close and significantly outperforms other methods. CCOT performs better on datasets 28, 29, 31, and 32, ECO stands out on dataset 38, and Siamese also performs well on datasets 22 and 36.

Prior Knowledge Regularized Multiview Self-Representation and its Applications.

To learn the self-representation matrices/tensor that encodes the intrinsic structure of the data, existing multiview self-representation models consider only the multiview features and, thus, impose equal membership preference across samples. However, this is inappropriate in real scenarios since the prior knowledge, e.g., explicit labels, semantic similarities, and weak-domain cues, can provide useful insights into the underlying relationship of samples. Based on this observation, this article proposes a prior knowledge regularized multiview self-representation (P-MVSR) model, in which the prior knowledge, multiview features, and high-order cross-view correlation are jointly considered to obtain an accurate self-representation tensor. The general concept of "prior knowledge" is defined as the complement of multiview features, and the core of P-MVSR is to take advantage of the membership preference, which is derived from the prior knowledge, to purify and refine the discovered membership of the data. Moreover, P-MVSR adopts the same optimization procedure to handle different prior knowledge and, thus, provides a unified framework for weakly supervised clustering and semisupervised classification. Extensive experiments on real-world databases demonstrate the effectiveness of the proposed P-MVSR model.

Continual Multiview Task Learning via Deep Matrix Factorization.

The state-of-the-art multitask multiview (MTMV) learning tackles a scenario where multiple tasks are related to each other via multiple shared feature views. However, in many real-world scenarios where a sequence of the multiview task comes, the higher storage requirement and computational cost of retraining previous tasks with MTMV models have presented a formidable challenge for this lifelong learning scenario. To address this challenge, in this article, we propose a new continual multiview task learning model that integrates deep matrix factorization and sparse subspace learning in a unified framework, which is termed deep continual multiview task learning (DCMvTL). More specifically, as a new multiview task arrives, DCMvTL first adopts a deep matrix factorization technique to capture hidden and hierarchical representations for this new coming multiview task while accumulating the fresh multiview knowledge in a layerwise manner. Then, a sparse subspace learning model is employed for the extracted factors at each layer and further reveals cross-view correlations via a self-expressive constraint. For model optimization, we derive a general multiview learning formulation when a new multiview task comes and apply an alternating minimization strategy to achieve lifelong learning. Extensive experiments on benchmark data sets demonstrate the effectiveness of our proposed DCMvTL model compared with the existing state-of-the-art MTMV and lifelong multiview task learning models.

On Reliable Multi-View Affinity Learning for Subspace Clustering

In multi-view subspace clustering, the low-rankness of the stacked self-representation tensor is widely accepted to capture the high-order cross-view correlation. However, using the nuclear norm as a convex surrogate of the rank function, the self-representation tensor exhibits strong connectivity with dense coefficients. When noise exists in the data, the generated affinity matrix may be unreliable for subspace clustering as it retains the connections across inter-cluster samples due to the lack of sparsity. Since both the connectivity and sparsity of the self-representation coefficients are curial for subspace clustering, we propose a Reliable Multi-View Affinity Learning (RMVAL) method so as to optimize both properties in a single model. Specifically, RMVAL employs the low-rank tensor constraint to yield a well-connected yet dense solution, and purifies the densely connected self-representation tensor by preserving only the connections in local neighborhoods using the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$l_1$</tex-math></inline-formula> -norm regularization. This way, the strong connections on the self-representation tensor are retained and the trivial coefficients corresponding to the inter-cluster connections are suppressed, leading to a “clean” self-representation tensor and also a reliable affinity matrix. We propose an efficient algorithm to solve RMVAL using the alternating direction method of multipliers. Extensive experiments on benchmark databases have demonstrated the superiority of RMVAL.

Multi-view subspace clustering via simultaneously learning the representation tensor and affinity matrix

Multi-view subspace clustering aims at separating data points into multiple underlying subspaces according to their multi-view features. Existing low-rank tensor representation-based multi-view subspace clustering algorithms are robust to noise and can preserve the high-order correlations of multi-view features. However, they may suffer from two common problems: (1) the local structures and different importance of each view feature are often neglected; (2) the low-rank representation tensor and affinity matrix are learned separately. To address these issues, we propose a unified framework to learn the Graph regularized Low-rank representation Tensor and Affinity matrix (GLTA) for multi-view subspace clustering. In the proposed GLTA framework, the tensor singular value decomposition-based tensor nuclear norm is adopted to explore the high-order cross-view correlations. The manifold regularization is exploited to preserve the local structures embedded in high-dimensional space. The importance of different features is automatically measured when constructing the final affinity matrix. An iterative algorithm is developed to solve GLTA using the alternating direction method of multipliers. Extensive experiments on seven challenging datasets demonstrate the superiority of GLTA over the state-of-the-art methods.

Cost-Sensitive Canonical Correlation Analysis for Semi-Supervised Multi-View Learning

Canonical correlation analysis (CCA) is a cost insensitive method. It assumes the same loss for different classification errors and aims to attain a low error rate by maximizing the cross-view correlation. However, in some real-world applications, different classification errors will lead to unequal misclassification losses. In addition, in practice, only limited cost label information is available in training set due to the expensive costs of labelling. This paper aims to perform label propagation with CCA in a unified cost-sensitive learning framework. By learning jointly, both the label propagation and CCA can feed back to each other. Thus, more discriminative and cost-sensitive projections will be learned for feature fusion. We test the proposed method on the cost-sensitive application of door-locker system based on multi-view face recognition. The results in comparison with eight label propagation methods, eleven CCA related methods and eight cost-sensitive single-view methods demonstrate its effectiveness.

Node-Sensitive Graph Fusion via Topo-Correlation for Image Retrieval

Various kinds of features prove to be effective for content-based image retrieval. However, due to the diversity of image contents, a descriptor may achieve impressive performance on specific images while becoming invalid on others. Although some efforts have been made to combine features as complementary counterparts, proper weighting scheme is still a challenge for fast and accurate retrieval. In this paper, we propose an effective fusion method, termed as Topo-correlation (Topo), where the importance of each feature is measured by cross-view correlations on local affinity graphs. Specifically, the weights of similarities are node-sensitive as well as modality-sensitive, thus boosting the results of good cues while depressing adverse factors for individual images. By estimating the consensus of similarity scores with regard to a query-driven criterion, the weighted graphs are generated efficiently with low computational complexity. Extensive experimental results on four benchmarks demonstrate the superiority of the proposed approach over the state-of-the-art methods.

Adversarial learning for multi-view network embedding on incomplete graphs

Network embedding, as a promising way of node representation learning, is capable of supporting various downstream network mining tasks, and has attracted growing research interests recently. Existing approaches mostly focus on learning the low-dimensional node representations by preserving the local or global topology information of a static network. It is difficult for such methods to learn desirable features for the nodes in an incomplete graph whose topology information is sparse or the new nodes in a dynamic graph. It is also challenging for them to deeply incorporate node attributes as the complementary information to improve the network embedding performance. To this end, in this paper we propose a Multi-View Adversarial learning based Network Embedding model named MVANE to deeply fuse the network topology information and node attributes to better perform network embedding on incomplete graphs. The insight is that the network topology and the node attributes are treated as two correlated views. The learned embedding vector of a node should be able to reveal its unique characteristics in both views. Specifically, the adversarial autoencoder is introduced as the basic model of MVANE. Autoencoder can learn a projection function to directly map the input feature vectors into the latent space, which ensures the MVANE learn embeddings for the new nodes through features projection without the need of retraining the model. Meanwhile, the adversarial learning strategy is also applied to better capture the cross-view correlations. The idea is that the learned embeddings in one view can not only reconstruct the inputs in this view, but also generate the features in another view. Under a unified learning framework, the latent representations in different views are fused and jointly reinforced by the proposed self/cross-view learning model. Empirically, we evaluate MVANE over multiple network datasets, and the results demonstrate the superiority of our proposal.

Graph regularized supervised cross-view hashing

Hashing methods have received significant attention for effective and efficient large scale similarity search in computer vision and information retrieval community. However, most existing cross-view hashing methods mainly focus on either similarity preservation of data or cross-view correlation. In this paper, we propose a graph regularized supervised cross-view hashing (GSCH) to preserve both the semantic correlation and the intra-view and inter view similarity simultaneously. In particular, GSCH uses intra-view similarity to estimate inter-view similarity structure. We further propose a sequential learning approach to derive the hashing function for each view. Experimental results on benchmark datasets against state-of-the-art methods show the effectiveness of our proposed method.

Joint intermodal and intramodal correlation preservation for semi-paired learning

Canonical correlation analysis(CCA) and its variants have been playing an important role in dimension reduction and information fusion. Most of the existing CCA-related algorithms can only handle fully-paired scenarios, where all the samples from different views require to be rigorously paired. However, majority of the samples only provide partial pairwise correspondences in most cases. In such semi-paired scenarios, traditional methods often perform poorly due to overfitting. Furthermore, most CCA-related algorithms only focus on the correlation between different views but ignore preserving the within-view similarity which is helpful to make samples in the same class mapped more compactly onto canonical correlation space. In addition, since CCA is unable to exploit discriminative information effectively, improved performance is limited in many tasks. To solve these challenging problems, we propose a novel method named joint Intermodal and Intramodal Semi-paired Canonical Correlation Analysis (I2SCCA), which properly handle the semi-paired scenarios. At the same time, the method preserves the within-view similarity information by making two neighbor samples in the same view closer in the projected space. Different from traditional similarity matrix construction, we incorporate a rank constraint of Laplacian matrix into the graph-based clustering algorithms. With fewer parameters, a graph that has exactly c (the number of clusters) connected components is learned without requiring any post graph-cut processing. Based on the obtained graph, we estimate the cross-view correlation through similarity measure of the single view. Specially, taking advantage of clustering assumption, discriminative information is derived although there is no label available. Finally, we extend I2SCCA to multi-view and non-linear cases. Extensive experiments on several datasets demonstrate the effectiveness of I2SCCA.

GII Representation-Based Cross-View Gait Recognition by Discriminative Projection With List-Wise Constraints

Remote person identification by gait is one of the most important topics in the field of computer vision and pattern recognition. However, gait recognition suffers severely from the appearance variance caused by the view change. It is very common that gait recognition has a high performance when the view is fixed but the performance will have a sharp decrease when the view variance becomes significant. Existing approaches have tried all kinds of strategies like tensor analysis or view transform models to slow down the trend of performance decrease but still have potential for further improvement. In this paper, a discriminative projection with list-wise constraints (DPLC) is proposed to deal with view variance in cross-view gait recognition, which has been further refined by introducing a rectification term to automatically capture the principal discriminative information. The DPLC with rectification (DPLCR) embeds list-wise relative similarity measurement among intraclass and inner-class individuals, which can learn a more discriminative and robust projection. Based on the original DPLCR, we have introduced the kernel trick to exploit nonlinear cross-view correlations and extended DPLCR to deal with the problem of multiview gait recognition. Moreover, a simple yet efficient gait representation, namely gait individuality image (GII), based on gait energy image is proposed, which could better capture the discriminative information for cross view gait recognition. Experiments have been conducted in the CASIA-B database and the experimental results demonstrate the outstanding performance of both the DPLCR framework and the new GII representation. It is shown that the DPLCR-based cross-view gait recognition has outperformed the-state-of-the-art approaches in almost all cases under large view variance. The combination of the GII representation and the DPLCR has further enhanced the performance to be a new benchmark for cross-view gait recognition.