Multiple Views Research Articles

DreamComposer++: Empowering Diffusion Models with Multi-View Conditions for 3D Content Generation.

Recent advancements in leveraging pre-trained 2D diffusion models achieve the generation of high-quality novel views from a single in-the-wild image. However, existing works face challenges in producing controllable novel views due to the lack of information from multiple views. In this paper, we present DreamComposer++, a flexible and scalable framework designed to improve current view-aware diffusion models by incorporating multi-view conditions. Specifically, DreamComposer++ utilizes a view-aware 3D lifting module to extract 3D representations of an object from various views. These representations are then aggregated and rendered into the latent features of target view through the multi-view feature fusion module. Finally, the obtained features of target view are integrated into pre-trained image or video diffusion models for novel view synthesis. Experimental results demonstrate that DreamComposer++ seamlessly integrates with cutting-edge view-aware diffusion models and enhances their abilities to generate controllable novel views from multi-view conditions. This advancement facilitates controllable 3D object reconstruction and enables a wide range of applications.

Simultaneous tracking of objects with loose context constraints from multiple views: human–human interaction paradigm

Simultaneous tracking of objects with loose context constraints from multiple views: human–human interaction paradigm

Pixel-Perfect Structure-From-Motion With Featuremetric Refinement.

Finding local features that are repeatable across multiple views is a cornerstone of sparse 3D reconstruction. The classical image matching paradigm detects keypoints per-image once and for all, which can yield poorly-localized features and propagate large errors to the final geometry. In this article, we refine two key steps of structure-from-motion by a direct alignment of low-level image information from multiple views: we first adjust the initial keypoint locations prior to any geometric estimation, and subsequently refine points and camera poses as a post-processing. This refinement is robust to large detection noise and appearance changes, as it optimizes a featuremetric error based on dense features predicted by a neural network. This significantly improves the accuracy of camera poses and scene geometry for a wide range of keypoint detectors, challenging viewing conditions, and off-the-shelf deep features. Our system easily scales to large image collections, enabling pixel-perfect crowd-sourced localization at scale.

Multiview Ensemble Learning Framework for Real‐Time UV Spectroscopic Detection of Nitrate in Water With Chemometric Modelling

ABSTRACTThe accuracy of detection of nitrate in water for quality monitoring is a significant yet challenging task. To address this, the present work proposes an ensemble machine learning–based chemometric framework for the optical detection of nitrate in water. It incorporates an absorbance‐based reagent‐less detection of nitrate in water to support the robustness of the model. The absorption spectra were recorded using a portable set‐up in the presence and absence of interfering ions. Different interfering ions, namely, nitrite (NO2−), calcium (Ca2+), magnesium (Mg2+), carbonate (CO32−), bromide (Br−), chloride (Cl−) and phosphate (PO43−), in all possible combinations (binary, ternary, quaternary, quinary, senary and septenary mixtures) are added to target analyte to validate the real‐time application of the proposed algorithm. Under the multiview framework, two models, MVNPM‐I and MVNPM‐II, i.e., multiview nitrate prediction models, are proposed. MVNPM‐I is based on an ensemble of regressors' results, and MVNPM‐II uses multiple views of the dataset followed by an ensemble of their results. The performance of the models is assessed using a hold‐out validation scheme with 10 repetitions and measured using R2 score and mean squared error (MSE). The best results of R2 score 0.9978 with a standard deviation 0.0014 and MSE of 1.1799 with a standard deviation of 0.8639 are obtained using the MVNPM‐II model. Further, the performance measures of the proposed models show that they can handle the presence of interfering ions. The algorithm was also tested using real‐world samples with an R2 score and MSE of 0.9998 and 0.696, respectively. The promising results strengthen the applicability of the proposed method in real‐world scenarios.

Multi-View Intrusion Detection Framework Using Deep Learning and Knowledge Graphs

Traditional intrusion detection systems (IDSs) rely on static rules and one-dimensional features, and they have difficulty dealing with zero-day attacks and highly concealed threats; furthermore, mainstream deep learning models cannot capture the correlation between multiple views of attacks due to their single perspective. This paper proposes a knowledge graph-enhanced multi-view deep learning framework, considering the strategy of integrating network traffic, host behavior, and semantic relationships; and evaluates the impact of the secondary fusion strategy on feature fusion to identify the optimal multi-view model configuration. The primary objective is to verify the superiority of multi-view feature fusion technology and determine whether incorporating knowledge graphs (KGs) can further enhance model performance. First, we introduce the knowledge graph (KG) as one of the feature views and neural networks as additional views, forming a multi-view feature fusion strategy that emphasizes the integration of spatial and relational features. The KG represents relational features combined with spatial features extracted by neural networks, enabling a more comprehensive representation of attack patterns through the synergy of both feature types. Secondly, based on this foundation, we propose a two-level fusion strategy. During the representation learning of spatial features, primary fusion is performed of each view, followed by secondary fusion with relational features from KG, thereby deepening and broadening feature integration. These strategies for understanding and deploying the multi-view concept improve the model’s expressive power and detection performance and also demonstrate strong generalization and robustness across three datasets, including TON_IoT and UNSW-NB15, marking a contribution of this study. After experimental evaluation, the F1 scores of multi-view models outperformed single-view models across all three datasets. Specifically, the F1 score of the multi-view approach (Model 6) improved by 10.57% on the TON_IoT Network+Win10 dataset compared with the best single-view model. In contrast, improvements of 5.53% and 3.21% were observed on the TON_IoT network and UNSW-NB15 datasets. In terms of feature fusion strategies, the secondary fusion strategy (Model 6) outperformed primary fusion (Model 5). Furthermore, incorporating KG-based relational features as a separate view improved model performance, a finding validated by ablation studies. Experimental results show that the deep fusion strategy of multi-dimensional data overcomes the limitations of traditional single-view models, enables collaborative multi-dimensional analysis of network attack behaviors, and significantly enhances detection capabilities in complex attack scenarios. This approach establishes a scalable multimodal analysis framework for intelligent cybersecurity, advancing intrusion detection beyond traditional rule-based methods toward semantic understanding.

Rotary Panoramic and Full-Depth-of-Field Imaging System for Pipeline Inspection.

To address the adaptability and insufficient imaging quality of conventional in-pipe imaging techniques for irregular pipelines or unstructured scenes, this study proposes a novel radial rotating full-depth-of-field focusing imaging system designed to adapt to the structural complexities of irregular pipelines, which can effectively acquire tiny details with a depth of 300-960 mm inside the pipeline. Firstly, a fast full-depth-of-field imaging method driven by depth features is proposed. Secondly, a full-depth rotating imaging apparatus is developed, incorporating a zoom camera, a miniature servo rotation mechanism, and a control system, enabling 360° multi-view angles and full-depth-of-field focusing imaging. Finally, full-depth-of-field focusing imaging experiments are carried out for pipelines with depth-varying characteristics. The results demonstrate that the imaging device can acquire depth data of the pipeline interior and rapidly obtain high-definition characterization sequence images of the inner pipeline wall. In the depth-of-field segmentation with multiple view angles, the clarity of the fused image is improved by 75.3% relative to a single frame, and the SNR and PSNR reach 6.9 dB and 26.3 dB, respectively. Compared to existing pipeline closed-circuit television (CCTV) and other in-pipeline imaging techniques, the developed rotating imaging system exhibits high integration, faster imaging capabilities, and adaptive capacity. This system provides an adaptive imaging solution for detecting defects on the inner surfaces of irregular pipelines, offering significant potential for practical applications in pipeline inspection and maintenance.

Anomaly detection in cropland monitoring using multiple view vision transformer

In recent times, the importance of low-altitude security, especially in agricultural surveillance, has seen a remarkable upswing. This paper puts forward a novel Internet of Drones framework tailored for low-altitude operations. Anomaly detection, which is pivotal for ensuring the integrity of the entire system, poses a substantial challenge. Such anomalies can range from unpredictable weather patterns in farmlands to unauthorized intrusions. To surmount this, a comprehensive deep learning pipeline is proposed in this study. It deploys a vision transformer model featuring a unique attention mechanism. The pipeline includes the meticulous collection of a vast array of normal and abnormal farmland images, followed by preprocessing to standardize data. Anomaly detection is then carried out, and the model’s performance is evaluated using metrics like sensitivity (92.8%), specificity (93.1%), accuracy (93.5%), and F1 score (94.1%). Comparative analysis with state-of-the-art algorithms reveals the superiority of the proposed model. In the future, this study plans to explore integrating data from thermal, infrared, or LIDAR sensors, enhance the interpretability of the vision transformer model, and optimize the deep learning pipeline to reduce computational complexity.

Balanced multi-view clustering with dynamic consistency exploration among multiple views

Balanced multi-view clustering with dynamic consistency exploration among multiple views

MUC: Mixture of Uncalibrated Cameras for Robust 3D Human Body Reconstruction

Multiple cameras can provide comprehensive multi-view video coverage of a person. Fusing this multi-view data is crucial for tasks like behavioral analysis, although it traditionally requires camera calibration—a process that is often complex. Moreover, previous studies have overlooked the challenges posed by self-occlusion under multiple views and the continuity of human body shape estimation. In this study, we introduce a method to reconstruct the 3D human body from multiple uncalibrated camera views. Initially, we utilize a pre-trained human body encoder to process each camera view individually, enabling the reconstruction of human body models and parameters for each view along with predicted camera positions. Rather than merely averaging the models across views, we develop a neural network trained to assign weights to individual views for all human body joints, based on the estimated distribution of joint distances from each camera. Additionally, we focus on the mesh surface of the human body for dynamic fusion, allowing for the seamless integration of facial expressions and body shape into a unified human body model. Our method has shown excellent performance in reconstructing the human body on two public datasets, advancing beyond previous work from the SMPL model to the SMPL-X model. This extension incorporates more complex hand poses and facial expressions, enhancing the detail and accuracy of the reconstructions. Crucially, it supports the flexible ad-hoc deployment of any number of cameras, offering significant potential for various applications.

Multi-view Consistent 3D Panoptic Scene Understanding

3D panoptic scene understanding seeks to create novel view images with 3D-consistent panoptic segmentation, which is crucial for many vision and robotics applications. Mainstream methods (e.g., Panoptic Lifting) directly use machine-generated 2D panoptic segmentation masks as training labels. However, these generated masks often exhibit multi-view inconsistencies, leading to ambiguities during the optimization process. To address this, we present Multi-view Consistent 3D Panoptic Scene Understanding (MVC-PSU), featuring two key components: 1) Probabilistic Semantic Aligner, which associates semantic information of corresponding pixels across multiple views by probabilistic alignment to ensure that predicted panoptic segmentation masks are consistent across different views. 2) Geometric Consistency Enforcer, which uses multi-view projection and monocular depth consistency to ensure that the geometry of the reconstructed scene is accurate and consistent across different views. Experimental results demonstrate that the proposed MVC-PSU surpasses state-of-the-art methods on the ScanNet, Replica, and HyperSim datasets.

3DHumanEdit: Multi-modal Body Part-aware Conditioning Information Integration for 3D Human Manipulation

The rapid advancement of 3D Generative Adversarial Networks (GANs) has significantly enhanced the diversity and quality of generated 3D images. Despite these breakthroughs, the manipulation capabilities of 3D GANs remain unexplored, presenting substantial challenges for practical applications where user interaction and modification are essential. Current manipulation methods often lack the precision needed for fine-grained attribute manipulation, and struggle to maintain multi-view consistency during the editing process. To address these limitations, we propose 3DHumanEdit, a novel approach for 3D human body part-aware manipulation. 3DHumanEdit leverages multi-modal feature fusion and body part-aware feature alignment to achieve precise manipulation of individual body parts based on detailed text inputs and segmentation images. By exploring 3D prior for accurate editing and enforcing correspondence in latent space, 3DHumanEdit ensures coherence across multiple views. Experiments demonstrate that 3DHumanEdit outperforms existing methods in both editing fidelity and multi-view consistency, offering a robust solution for fine-grained 3D manipulation.

Disentangled Contrastive Bundle Recommendation with Conditional Diffusion

Bundle recommendation aims to improve user experience by suggesting complementary items that users are likely to purchase together. Although recent advances in recommendation systems have shown promise, there are still significant challenges: i) The dynamic nature of user preferences and interactions introduces noise that can distort the effectiveness of recommendations. ii) Existing methods frequently exhibit limited robustness when addressing the sparsity of user interactions with bundles in real-world scenarios. To tackle these issues, we introduce a disentangled contrastive bundle recommendation (DCBR) framework with conditional diffusion. First, we propose a conditional bundle diffusion model for denoising the user-bundle interaction graph, introducing a bundle latent consistency constraint during the optimization process to mitigate the degradation of original interaction information. Subsequently, we design a triple-view denoised graph learning module to obtain effective representations from multiple views. Furthermore, we present a dual-level disentangled contrastive learning paradigm, which addresses the latent relationships at two levels: between views (inter-view) and within each view (intra-view). By maximizing the consistency between positive samples in these contrastive views, we generate disentangled contrastive signals, overcoming interaction sparsity and alleviating noise issues. Our experimental evaluations on three benchmark datasets reveal that DCBR significantly outperforms state-of-the-art methods.

Scalable One-Pass Incomplete Multi-View Clustering by Aligning Anchors

Multi-view clustering has gained increasing attention by utilizing the complementary and consensus information across views. To alleviate the computation cost for the existing multi-view clustering approaches on datasets with large scales, studies based on anchor have been presented. Although extensively adopted in the real scenarios, most of these works ignore to learn an integral subspace revealing the cluster structure with anchors from different views being aligned, where the centroid and cluster assignment matrix can be directly achieved based on the integral subspace. Moreover, these works neglect to perform the alignment among anchors and integral subspace learning in a unified model on the incomplete multi-view dataset. Then the mutual improvements among aligning anchors and learning integral subspace are not guaranteed in optimizing the objective function, which inevitably limit the representation ability of the model and result in the suboptimal clustering performance. In this paper, we propose a novel anchor learning method for incomplete multi-view dataset termed Scalable One-pass incomplete Multi-view clustEring by Aligning anchorS (SOME-AS). Specifically, we capture the complementary information among multiple views by building the anchor graph for each view on the incomplete dataset. The integral subspace reflecting the cluster structure is learned with the alignment among anchors from different views being considered. We build the cluster assignment and centroid representation with orthogonal constraint to approximate the integral subspace. Then the subspace itself and the partition are simultaneously taken into account in this manner. Besides, the mutual improvements among aligning anchors and learning integral subspace are able to be ensured. Experiments on several incomplete multi-view datasets validate the efficiency and effectiveness of SOME-AS.

MV-VTON: Multi-View Virtual Try-On with Diffusion Models

The goal of image-based virtual try-on is to generate an image of the target person naturally wearing the given clothing. However, existing methods solely focus on the frontal try-on using the frontal clothing. When the views of the clothing and person are significantly inconsistent, particularly when the person's view is non-frontal, the results are unsatisfactory. To address this challenge, we introduce Multi-View Virtual Try-ON (MV-VTON), which aims to reconstruct the dressing results from multiple views using the given clothes. Given that single-view clothes provide insufficient information for MV-VTON, we instead employ two images, i.e., the frontal and back views of the clothing, to encompass the complete view as much as possible. Moreover, we adopt diffusion models that have demonstrated superior abilities to perform our MV-VTON. In particular, we propose a view-adaptive selection method where hard-selection and soft-selection are applied to the global and local clothing feature extraction, respectively. This ensures that the clothing features are roughly fit to the person's view. Subsequently, we suggest joint attention blocks to align and fuse clothing features with person features. Additionally, we collect a MV-VTON dataset MVG, in which each person has multiple photos with diverse views and poses. Experiments show that the proposed method not only achieves state-of-the-art results on MV-VTON task using our MVG dataset, but also has superiority on frontal-view virtual try-on task using VITON-HD and DressCode datasets.

SVTformer: Spatial-View-Temporal Transformer for Multi-View 3D Human Pose Estimation

Recently, transformer-based methods have been introduced to estimate 3D human pose from multiple views by aggregating the spatial-temporal information of human joints to achieve the lifting of 2D to 3D. However, previous approaches cannot model the inter-frame correspondence of each view's joint individually, nor can they directly consider all view interactions at each time, leading to insufficient learning of multi-view associations. To address this issue, we propose a Spatial-View-Temporal transformer (SVTformer) to decouple spatial-view-temporal information in sequential order for correlation learning and model dependencies between them in a local-to-global manner. SVTformer includes an attended Spatial-View-Temporal (SVT) patch embedding to attentively capture the local features of the input poses and stacked SVT encoders to extract global spatial-view-temporal dependencies progressively. Specifically, SVT encoders perform three reconstructions sequentially to attended features with the learning through view decoupling for temporal-enhanced spatial correlation, temporal decoupling for spatial-enhanced view correlation, and another view decoupling for spatial-enhanced temporal relationship. This decoupling-coupling-decoupling multi-view scheme enables us to alternatively model the inter-joint spatial relationships, cross-view dependencies, and temporal motion associations. We evaluate the proposed SVTformer on three popular 3D HPE datasets, and it yields state-of-the-art performance. It effectively deals with ill-posed problems and enhances the accuracy of 3D human pose estimation.

Noisy Label Calibration for Multi-View Classification

In recent years, multi-view learning has aroused extensive research passion. Most existing multi-view learning methods often rely on well-annotations to improve decision accuracy. However, noise labels are ubiquitous in multi-view data due to imperfect annotations. To deal with this problem, we propose a novel noisy label calibration method (NLC) for multi-view classification to resist the negative impact of noisy labels. Specifically, to capture consensus information from multiple views, we employ max-margin rank loss to reduce the heterogeneous gap. Subsequently, we evaluate the confidence scores to enrich predictions associated with noise instances according to all reliable neighbors. Further, we propose Label Noise Detection (LND) to separate multi-view data into a clean or noisy subset, and propose Label Calibration Learning (LCL) to correct noisy instances. Finally, we adopt the cross-entropy loss to achieve multi-view classification. Extensive experiments on six datasets validate that our method outperforms eight state-of-the-art methods.

Multi-View 3D Human Pose Estimation with Weakly Synchronized Images

Multi-view 3D human pose estimation (MHPE) is an important research task in computer vision. To maintain consistency during the data collection, hardware synchronization devices are commonly used to connect cameras, ensuring that images from different views are captured simultaneously. However, synchronizing with extra devices has two apparent limitations: the hardware is i) usually expensive and ii) less flexible for deployment in outdoor open scenarios. Suppose the model can improve its tolerance for the time differences in multi-view image capture. In that case, the difficulty and cost of deployment will be greatly reduced, and MHPE will become more widespread. In this paper, we try to answer how to build a model that performs pose estimation directly using ''weakly synchronized images" from multiple views, where the captured images shift from each other within a frame. To this end, we introduce a new multi-view 3D human pose estimation task given weakly synchronized image inputs. Apart from existing well-synchronized datasets, we present the first weakly synchronized dataset comprising 800k images. Thereon, we propose SyncDiffPose, a novel model based on the diffusion method for pose estimation to denoise the error in such data. By combining simple synchronization strategies, e.g., the timer method, our approach can perform pose estimation without hardware calibration.

CLIP-driven View-aware Prompt Learning for Unsupervised Vehicle Re-identification

With the emergence of vision-language pre-trained models, such as CLIP, some textual prompts have been gradually introduced recently into re-identification (Re-ID) tasks to obtain considerably robust multimodal information. However, most textual descriptions based on vehicle Re-ID tasks only contain identity index words without specific words to describe vehicle view information, thereby resulting in difficulty to be widely applied in vehicle Re-ID tasks with view variations. This case inspires us to propose a CLIP-driven view-aware prompt learning framework for unsupervised vehicle Re-ID. We first design a learnable textual prompt template called view-aware context optimization (ViewCoOp) based on dynamic multi-view word embeddings, which can fully obtain the proportion and position encoding of each view in the whole vehicle body region. Subsequently, a cross-modal mutual graph is constructed to explore the connections between inter-modal and intra-modal. Each sample is treated as a graph node, which extracts textual features based on ViewCoOp and the visual features of images. Moreover, leveraging the inter-cluster and intra-cluster correlation in the bimodal clustering results in the determination of connectivity between graph node pairs. Lastly, the proposed cross-modal mutual graph method utilizes supervised information from the bimodal gap to directly fine-tune the image encoder of CLIP for downstream unsupervised vehicle Re-ID tasks. Extensive experiments verify that the proposed method is capable of effectively obtaining cross-modal description ability from multiple views.

Multi-View Unsupervised Column Subset Selection via Combinatorial Search (Student Abstract)

Given a data matrix, unsupervised column subset selection refers to the problem of identifying a subset of columns that can be used to linearly approximate the original data matrix. This problem has many applications, such as feature selection and representative selection, but solving it optimally is known to be NP-hard. We consider multi-view unsupervised column subset selection, which extends the concept of (single-view) column subset selection to data represented in multiple views or modalities. We introduce a combinatorial search algorithm for this generalized problem. One variant of the algorithm is guaranteed to compute an optimal solution in a setting similar to the classical A* algorithm. Other suboptimal variants, in a setting similar to the weighted A* algorithm, are much faster and provide a solution along with a bound on its quality.

NightReID: A Large-Scale Nighttime Person Re-Identification Benchmark

Person re-identification (Re-ID) is crucial for intelligent surveillance systems, facilitating the identification of individuals across multiple camera views. While significant advancements have been made for daytime scenarios, ensuring reliable Re-ID performance during nighttime remains a significant challenge. Given the cost and limited accessibility of infrared cameras, we investigate a critical question: Can RGB cameras be effectively utilized for accurate Re-ID during nighttime? To address this, we introduce NightReID, a large-scale RGB Re-ID dataset collected from a real-world nighttime surveillance system. NightReID includes 1,500 identities and over 53,000 images, capturing diverse scenes with complex lighting and adverse weather conditions. This rich dataset provides a valuable benchmark for advancing nighttime Re-ID research. Moreover, we propose the Enhancement, Denoising, and Alignment (EDA) framework with two novel modules to enhance nighttime Re-ID performance. First, an unsupervised Image Enhancement and Denoising (IED) method is designed to improve the quality of nighttime images, preserving critical details while removing noise without requiring paired ground truth. Second, we introduce Data Distribution Alignment (DDA) through statistical priors, aligning the distributions between pre-training data and nighttime data to mitigate domain shift. Extensive experiments on multiple nighttime Re-ID datasets demonstrate the significance of NightReID and validate the efficacy, flexibility, and applicability of the EDA framework.