Multi-view Images Research Articles

A multi-task deep learning approach for real-time view classification and quality assessment of echocardiographic images

High-quality standard views in two-dimensional echocardiography are essential for accurate cardiovascular disease diagnosis and treatment decisions. However, the quality of echocardiographic images is highly dependent on the practitioner’s experience. Ensuring timely quality control of echocardiographic images in the clinical setting remains a significant challenge. In this study, we aimed to propose new quality assessment criteria and develop a multi-task deep learning model for real-time multi-view classification and image quality assessment (six standard views and “others”). A total of 170,311 echocardiographic images collected between 2015 and 2022 were utilized to develop and evaluate the model. On the test set, the model achieved an overall classification accuracy of 97.8% (95%CI 97.7–98.0) and a mean absolute error of 6.54 (95%CI 6.43–6.66). A single-frame inference time of 2.8 ms was achieved, meeting real-time requirements. We also analyzed pre-stored images from three distinct groups of echocardiographers (junior, senior, and expert) to evaluate the clinical feasibility of the model. Our multi-task model can provide objective, reproducible, and clinically significant view quality assessment results for echocardiographic images, potentially optimizing the clinical image acquisition process and improving AI-assisted diagnosis accuracy.

Wear particles image enhancement using long short-term memory 3D recurrent reconstruction neural network (LSTM 3D-R2N2)

3D modeling of wear particles has proven to be a useful tool for monitoring mechanical failure conditions. In this work, a new method for 3D reconstruction of wear particles in uncontaminated oil (healthy oil) and contaminated oil (used oil) was proposed. The image acquisition device can capture multi-view images of moving wear particles in both healthy and used oil by using the reflected light. The images were pretreated first, and the image color inversion was conducted using the Pillow library. The pretreated wear particle images were used for 3D reconstruction using long short-term memory 3D recurrent reconstruction neural network. The current results were verified against existing results, and good agreement can be found. It can be concluded that we can reconstruct the similar 3D wear particle results with fewer images by comparison with other methods. Specifically, only 4–6 image samples were used for the 3D reconstruction of wear particles, and at least 8 image samples were needed for other existing reports.

NeRF-Texture: Synthesizing Neural Radiance Field Textures.

Texture synthesis is a fundamental problem in computer graphics that would benefit various applications. Existing methods are effective in handling 2D image textures. In contrast, many real-world textures contain meso-structure in the 3D geometry space, such as grass, leaves, and fabrics, which cannot be effectively modeled using only 2D image textures. We propose a novel texture synthesis method with Neural Radiance Fields (NeRF) to capture and synthesize textures from given multi-view images. In the proposed NeRF texture representation, a scene with fine geometric details is disentangled into the meso-structure textures and the underlying base shape. This allows textures with meso-structure to be effectively learned as latent features situated on the base shape, which are fed into a NeRF decoder trained simultaneously to represent the rich view-dependent appearance. Using this implicit representation, we can synthesize NeRF-based textures through patch matching of latent features. However, inconsistencies between the metrics of the reconstructed content space and the latent feature space may compromise the synthesis quality. To enhance matching performance, we further regularize the distribution of latent features by incorporating a clustering constraint. In addition to generating NeRF textures over a planar domain, our method can also synthesize NeRF textures over curved surfaces, which are practically useful. Experimental results and evaluations demonstrate the effectiveness of our approach.

Neural 3D Scene Reconstruction With Indoor Planar Priors.

This paper addresses the challenge of reconstructing 3D indoor scenes from multi-view images. Many previous works have shown impressive reconstruction results on textured objects, but they still have difficulty in handling low-textured planar regions, which are common in indoor scenes. An approach to solving this issue is to incorporate planar constraints into the depth map estimation in multi-view stereo-based methods, but the per-view plane estimation and depth optimization lack both efficiency and multi-view consistency. In this work, we show that the planar constraints can be conveniently integrated into the recent implicit neural representation-based reconstruction methods. Specifically, we use an MLP network to represent the signed distance function as the scene geometry. Based on the Manhattan-world assumption and the Atlanta-world assumption, planar constraints are employed to regularize the geometry in floor and wall regions predicted by a 2D semantic segmentation network. To resolve the inaccurate segmentation, we encode the semantics of 3D points with another MLP and design a novel loss that jointly optimizes the scene geometry and semantics in 3D space. Experiments on ScanNet and 7-Scenes datasets show that the proposed method outperforms previous methods by a large margin on 3D reconstruction quality.

Prior-free 3D human pose estimation in a video using limb-vectors

Estimating accurate 3D human poses from a monocular video is fundamental to various computer vision tasks. Existing methods exploit 2D-to-3D pose lifting, multiview images, and depth sensors to model spatio-temporal dependencies. However, depth ambiguities, occlusions, and larger temporal receptive fields pose challenges to these approaches. To address this, we propose a novel prior-free DCNN-based 3D human pose estimation method for monocular image sequences using limb vectors. Our method comprises two subnetworks: a limb direction estimator and a limb length estimator. The limb direction estimator utilizes a fully convolutional network to model limb direction vectors across a temporal window. We show that network complexity can be significantly reduced by utilizing dilated convolutional operations and a relatively smaller receptive field while maintaining estimation accuracy. Moreover, the limb length estimator captures stable limb length estimations from a reliable frame set. Our model has shown superior performance compared to existing methods on the Human3.6M and MPI-INF-3DHP datasets.

Development of multi-view 3D reconstruction system for bubble flow measurement

This work introduces the development of bubble measurement method utilizing a three-dimensional (3D) reconstruction technique from multi-view images. Multiple synchronized cameras were positioned around a water container, and calibration was performed to obtain external and internal camera parameters. Images of bubbles emerging from a nozzle were captured, and a machine learning technique was used to extract bubble silhouettes as foreground probability distributions, enabling the extraction of bubbles from images obtained with a simple lighting setup. These distributions were then projected onto a 3D voxel space using the visual hull method. It was confirmed that the method can successfully capture bubble generation, detachment, and rise behaviors, offering insights into understanding bubble dynamics and potential applications in 3D computational fluid dynamics validation.

Digital image correlation calculation method for RGB-D camera multi-view matching using variable template

Multi-view matching is an indispensable method for digital image correlation 3D reconstruction. However, as the camera viewing angle changes, traditional DIC methods can lead to the accumulation of matching errors and an increase in mismatches, thereby affecting the 3D reconstruction quality. This paper proposes a digital image correlation method for multi-view RGB-D images based on variable template matching. Initial positioning is achieved by dynamically adjusting the size and sampling direction of the variable template. Sub-pixel registration is then performed using the inverse compositional algorithm, followed by multi-view point cloud fusion using the Iterative Closest Point (ICP) registration method. Experimental results show that the iteration speed of point cloud registration is increased by 30%, and the cumulative error is reduced by 62.5%. These improvements demonstrate that the proposed method is suitable for multi-viewpoint matching scenarios, achieving excellent results in multi-view feature point matching and multi-view point cloud fusion, significantly enhancing 3D reconstruction accuracy and efficiency.

Multi-view scene matching with relation aware feature perception

For scene matching, the extraction of metric features is a challenging task in the face of multi-source and multi-view scenes. Aiming at the requirements of multi-source and multi-view scene matching, a siamese network model for Spatial Relation Aware feature perception and fusion is proposed. The key contributions of this work are as follows: (1) Seeking to enhance the coherence of multi-view image features, we investigate the relation aware feature perception. With the help of spatial relation vector decomposition, the distribution consistency perception of image features in the horizontal H→ and vertical W→ directions is realized. (2) In order to establish the metric consistency relationship, the large-scale local information perception strategy is studied to realize the relative trade-off scale selection under the size of mainstream aerial images and satellite images. (3) After obtaining the multi-scale metric features, in order to improve the metric confidence, the feature selection and fusion strategy is proposed. The significance of distinct feature levels in the backbone network is systematically assessed prior to fusion, leading to an enhancement in the representation of pivotal components within the metric features during the fusion process. The experimental results obtained from the University-1652 dataset and the collected real scene data affirm the efficacy of the proposed method in enhancing the reliability of the metric model. The demonstrated effectiveness of this method suggests its applicability to diverse scene matching tasks.

Automated Measurement of Cattle Dimensions Using Improved Keypoint Detection Combined with Unilateral Depth Imaging.

Traditional measurement methods often rely on manual operations, which are not only inefficient but also cause stress to cattle, affecting animal welfare. Currently, non-contact cattle dimension measurement usually involves the use of multi-view images combined with point cloud or 3D reconstruction technologies, which are costly and less flexible in actual farming environments. To address this, this study proposes an automated cattle dimension measurement method based on an improved keypoint detection model combined with unilateral depth imaging. Firstly, YOLOv8-Pose is selected as the keypoint detection model and SimSPPF replaces the original SPPF to optimize spatial pyramid pooling, reducing computational complexity. The CARAFE architecture, which enhances upsampling content-aware capabilities, is introduced at the neck. The improved YOLOv8-pose achieves a mAP of 94.4%, a 2% increase over the baseline model. Then, cattle keypoints are captured on RGB images and mapped to depth images, where keypoints are optimized using conditional filtering on the depth image. Finally, cattle dimension parameters are calculated using the cattle keypoints combined with Euclidean distance, the Moving Least Squares (MLS) method, Radial Basis Functions (RBFs), and Cubic B-Spline Interpolation (CB-SI). The average relative errors for the body height, lumbar height, body length, and chest girth of the 23 measured beef cattle were 1.28%, 3.02%, 6.47%, and 4.43%, respectively. The results show that the method proposed in this study has high accuracy and can provide a new approach to non-contact beef cattle dimension measurement.

Indoor Scene Reconstruction with Fine-Grained Details Using Hybrid Representation and Normal Prior Enhancement.

The reconstruction of indoor scenes from multi-view RGB images is challenging due to the coexistence of flat and texture-less regions alongside delicate and fine-grained regions. Recent methods leverage neural radiance fields aided by predicted surface normal priors to recover the scene geometry. These methods excel in producing complete and smooth results for floor and wall areas. However, they struggle to capture complex surfaces with high-frequency structures due to the inadequate neural representation and the inaccurately predicted normal priors. This work aims to reconstruct high-fidelity surfaces with fine-grained details by addressing the above limitations. To improve the capacity of the implicit representation, we propose a hybrid architecture to represent low-frequency and high-frequency regions separately. To enhance the normal priors, we introduce a simple yet effective image sharpening and denoising technique, coupled with a network that estimates the pixel-wise uncertainty of the predicted surface normal vectors. Identifying such uncertainty can prevent our model from being misled by unreliable surface normal supervisions that hinder the accurate reconstruction of intricate geometries. Experiments on the benchmark datasets show that our method outperforms existing methods in terms of reconstruction quality. Furthermore, the proposed method also generalizes well to real-world indoor scenarios captured by our hand-held mobile phones. Our code is publicly available at: https://github.com/yec22/Fine-Grained-Indoor-Recon.

NeRF-Art: Text-Driven Neural Radiance Fields Stylization.

As a powerful representation of 3D scenes, the neural radiance field (NeRF) enables high-quality novel view synthesis from multi-view images. Stylizing NeRF, however, remains challenging, especially in simulating a text-guided style with both the appearance and the geometry altered simultaneously. In this paper, we present NeRF-Art, a text-guided NeRF stylization approach that manipulates the style of a pre-trained NeRF model with a simple text prompt. Unlike previous approaches that either lack sufficient geometry deformations and texture details or require meshes to guide the stylization, our method can shift a 3D scene to the target style characterized by desired geometry and appearance variations without any mesh guidance. This is achieved by introducing a novel global-local contrastive learning strategy, combined with the directional constraint to simultaneously control both the trajectory and the strength of the target style. Moreover, we adopt a weight regularization method to effectively suppress cloudy artifacts and geometry noises which arise easily when the density field is transformed during geometry stylization. Through extensive experiments on various styles, we demonstrate that our method is effective and robust regarding both single-view stylization quality and cross-view consistency.

RIS-Aided Single-Frequency 3D Imaging by Exploiting Multi-View Image Correlations

RIS-Aided Single-Frequency 3D Imaging by Exploiting Multi-View Image Correlations

3D Human Pose Estimation Based on Wearable IMUs and Multiple Camera Views

The problem of 3D human pose estimation (HPE) has been the focus of research in recent years, yet precise estimation remains an under-explored challenge. In this paper, the merits of both multiview images and wearable IMUs are combined to enhance the process of 3D HPE. We build upon a state-of-the-art baseline while introducing three novelties. Initially, we enhance the precision of keypoint localization by substituting Gaussian kernels with Laplacian kernels in the generation of target heatmaps. Secondly, we incorporate orientation regularized network (ORN), which enhances cross-modal heatmap fusion by taking a weighted average of the top-scored values instead of solely relying on the maximum value. This not only improves robustness to outliers but also leads to higher accuracy in pose estimation. Lastly, we modify the limb length constraint in the conventional orientation regularized pictorial structure model (ORPSM) to improve the estimation of joint positions. Specifically, we devise a soft-coded binary term for limb length constraint, hence imposing a flexible and smoothed penalization and reducing sensitivity to hyperparameters. The experimental results using the TotalCapture dataset reveal a significant improvement, with a 10.3% increase in PCKh accuracy at the one-twelfth threshold and a 3.9 mm reduction in MPJPE error compared to the baseline.

Improved mirror-assisted multi-view digital image correlation for accuracy-enhanced dual-surface profile and deformation measurement

Accurate measurement of dual-surface kinematic fields is essential for characterizing the mechanical behavior and anisotropic parameters of sheet specimens. To this end, a cost-efficient and simple-to-implement mirror-assisted multi-view digital image correlation (MV-DIC) was recently proposed and has been proven to be effective. Nevertheless, this method conventionally relies on reflection transformation calibration to map the measured virtual surfaces to their actual positions. The positional transformation for each surface is mutually independent and lacks constraint relationships. This work proposes an improved mirror-assisted MV-DIC method for accuracy-enhanced dual-surface deformation measurement by utilizing the overlapped region of the virtual surfaces instead of reflection transformation calibration. Through preliminary feature matching and precise matching of subsets for corresponding points in the overlapped regions, the relative positional transformations between the measured surfaces of the specimen can be accurately determined. As the accuracy of positional transformations primarily depends on the matching accuracy of corresponding points rather than the accuracy of multiple plane fitting, this method provides better relative positional relationships between the two surfaces compared to conventional methods. The results of several real experiments well validated the practicality and accuracy of the proposed method.

A vision measurement method for ship hull plates based on multi-view stereo and image segmentation

The background denoising of point clouds is challenging for the measurement of ship hull plates. To address this issue, a vision measurement method is proposed by combining an identity cost volume network (ICV-Net) and a segment anything model (SAM). By applying the ship hull plate masks segmented by SAM to the corresponding depth maps inferred by ICV-Net, the background noise can be directly removed at the depth map level. Additionally, an edge extraction optimization algorithm complementary to SAM is proposed to eliminate the thickness-related noise. Moreover, an efficient depth map refinement algorithm is proposed by imposing the smoothness prior of ship hull plate surfaces and multi-view photometric consistency. Sub-pixel accurate depth maps of ship hull plates can be obtained by the depth map refinement. Experimental results suggest that the proposed method can automatically, accurately and efficiently remove the background noise of ship hull plates, which is crucial toward practical application.

Multimodal spatiotemporal aggregation for point cloud accumulation

Point cloud accumulation is a crucial technique in point cloud analysis, facilitating various downstream tasks like surface reconstruction. Current methods merely rely on raw LiDAR points, yielding unsatisfactory performance due to the limited geometric information, particularly in complex scenarios characterized by intricate motions, diverse objects, and an increased number of frames. In this paper, we introduce camera modality data, which is usually acquired alongside LiDAR data at minimal expense. To this end, we present the Multimodal Spatiotemporal Aggregation solution (termed MSA) to thoroughly explore and aggregate these two distinct modalities (sparse 3D points and multi-view 2D images). Concretely, we propose a multimodal spatial aggregation module to bridge the data gap between different modalities in the Bird’s-Eye-View (BEV) space and further fuse them by learnable adaptive channel-wise weights. By assembling their respective strengths, this module generates a reliable and consistent scene representation. Subsequently, we design a temporal aggregation module to capture continuous motion information across consecutive sequences, which is beneficial for identifying the motion state of the foreground scene and enabling the model to extend robustly to longer sequences. Experiments demonstrate MSA outperforms state-of-the-art (SoTA) point cloud accumulation methods across all evaluation metrics in the public benchmark, especially with more frames.

NeuralTO: Neural Reconstruction and View Synthesis of Translucent Objects

Learning from multi-view images using neural implicit signed distance functions shows impressive performance on 3D Reconstruction of opaque objects. However, existing methods struggle to reconstruct accurate geometry when applied to translucent objects due to the non-negligible bias in their rendering function. To address the inaccuracies in the existing model, we have reparameterized the density function of the neural radiance field by incorporating an estimated constant extinction coefficient. This modification forms the basis of our innovative framework, which is geared towards highfidelity surface reconstruction and the novel-view synthesis of translucent objects. Our framework contains two stages. In the reconstruction stage, we introduce a novel weight function to achieve accurate surface geometry reconstruction. Following the recovery of geometry, the second phase involves learning the distinct scattering properties of the participating media to enhance rendering. A comprehensive dataset, comprising both synthetic and real translucent objects, has been built for conducting extensive experiments. Experiments reveal that our method outperforms existing approaches in terms of reconstruction and novel-view synthesis.

SketchDream: Sketch-based Text-To-3D Generation and Editing

Existing text-based 3D generation methods generate attractive results but lack detailed geometry control. Sketches, known for their conciseness and expressiveness, have contributed to intuitive 3D modeling but are confined to producing texture-less mesh models within predefined categories. Integrating sketch and text simultaneously for 3D generation promises enhanced control over geometry and appearance but faces challenges from 2D-to-3D translation ambiguity and multi-modal condition integration. Moreover, further editing of 3D models in arbitrary views will give users more freedom to customize their models. However, it is difficult to achieve high generation quality, preserve unedited regions, and manage proper interactions between shape components. To solve the above issues, we propose a text-driven 3D content generation and editing method, SketchDream, which supports NeRF generation from given hand-drawn sketches and achieves free-view sketch-based local editing. To tackle the 2D-to-3D ambiguity challenge, we introduce a sketch-based multi-view image generation diffusion model, which leverages depth guidance to establish spatial correspondence. A 3D ControlNet with a 3D attention module is utilized to control multi-view images and ensure their 3D consistency. To support local editing, we further propose a coarse-to-fine editing approach: the coarse phase analyzes component interactions and provides 3D masks to label edited regions, while the fine stage generates realistic results with refined details by local enhancement. Extensive experiments validate that our method generates higher-quality results compared with a combination of 2D ControlNet and image-to-3D generation techniques and achieves detailed control compared with existing diffusion-based 3D editing approaches.

CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets

In the realm of digital creativity, our potential to craft intricate 3D worlds from imagination is often hampered by the limitations of existing digital tools, which demand extensive expertise and efforts. To narrow this disparity, we introduce CLAY, a 3D geometry and material generator designed to effortlessly transform human imagination into intricate 3D digital structures. CLAY supports classic text or image inputs as well as 3D-aware controls from diverse primitives (multi-view images, voxels, bounding boxes, point clouds, implicit representations, etc). At its core is a large-scale generative model composed of a multi-resolution Variational Autoencoder (VAE) and a minimalistic latent Diffusion Transformer (DiT), to extract rich 3D priors directly from a diverse range of 3D geometries. Specifically, it adopts neural fields to represent continuous and complete surfaces and uses a geometry generative module with pure transformer blocks in latent space. We present a progressive training scheme to train CLAY on an ultra large 3D model dataset obtained through a carefully designed processing pipeline, resulting in a 3D native geometry generator with 1.5 billion parameters. For appearance generation, CLAY sets out to produce physically-based rendering (PBR) textures by employing a multi-view material diffusion model that can generate 2K resolution textures with diffuse, roughness, and metallic modalities. We demonstrate using CLAY for a range of controllable 3D asset creations, from sketchy conceptual designs to production ready assets with intricate details. Even first time users can easily use CLAY to bring their vivid 3D imaginations to life, unleashing unlimited creativity.

CharacterGen: Efficient 3D Character Generation from Single Images with Multi-View Pose Canonicalization

BNRist, Department of Computer Science and Technology, Tsinghua University, China In the field of digital content creation, generating high-quality 3D characters from single images is challenging, especially given the complexities of various body poses and the issues of self-occlusion and pose ambiguity. In this paper, we present CharacterGen, a framework developed to efficiently generate 3D characters. CharacterGen introduces a streamlined generation pipeline along with an image-conditioned multi-view diffusion model. This model effectively calibrates input poses to a canonical form while retaining key attributes of the input image, thereby addressing the challenges posed by diverse poses. A transformer-based, generalizable sparse-view reconstruction model is the other core component of our approach, facilitating the creation of detailed 3D models from multi-view images. We also adopt a texture-back-projection strategy to produce high-quality texture maps. Additionally, we have curated a dataset of anime characters, rendered in multiple poses and views, to train and evaluate our model. Our approach has been thoroughly evaluated through quantitative and qualitative experiments, showing its proficiency in generating 3D characters with high-quality shapes and textures, ready for downstream applications such as rigging and animation.