Single-view Image Research Articles

NP-Hand: Novel Perspective Hand Image Synthesis Guided by Normals.

Synthesizing multi-view images that are geometrically consistent with a given single-view image is one of the hot issues in AIGC in recent years. Existing methods have achieved impressive performance on objects with symmetry or rigidity, but they are inappropriate for the human hand. Because an image-captured human hand has more diverse poses and less attractive textures. In this paper, we propose NP-Hand, a framework that elegantly combines the diffusion model and generative adversarial network: The multi-step diffusion is trained to synthesize low-resolution novel perspective, while the single-step generator is exploited to further enhance synthesis quality. To maintain the consistency between inputs and synthesis, we creatively introduce normal maps into NP-Hand to guide the whole synthesizing process. Comprehensive evaluations have demonstrated that the proposed framework is superior to existing state-of-the-art models and more suitable for synthesizing hand images with faithful structures and realistic appearance details. The code will be released on our website.

ProtoCar: Learning 3D Vehicle Prototypes from Single-View and Unconstrained Driving Scene Images

Reconstructing 3D models from sensor data is a valuable and promising direction for developing testing and validation environments in applications like autonomous driving. However, existing methods for 3D modeling often rely on extensive multi-view data or controlled conditions, making them difficult and expensive to scale. Furthermore, these methods, particularly those based on neural radiance fields, typically produce implicit models that can be challenging to manipulate and suffer from slow rendering speeds. In this paper, we introduce ProtoCar, a novel approach that overcomes these limitations by learning 3D vehicle prototypes from single-view images with diverse and unconstrained visual conditions. ProtoCar uses real-world driving data from LiDAR and image sensors, and employs 3D Gaussian splatting techniques to represent explicit geometric and texture. Extensive experiments demonstrate that ProtoCar generates high-quality 3D models and adapts well to various vehicle types and challenging visual scenarios, offering a scalable and effective solution for 3D modeling in environments with limited and variable visual information.

Single-view Image to Novel-view Generation for Hand-Object Interactions

Hand-object interaction modeling from a single RGB image is a significantly challenging task. Previous works typically reconstruct hand-object interactions as texture-less meshes, ignoring photo-realistic image generation. In this work, we introduce the HO123, a novel method to synthesize novel-view hand-object interaction images from a single image. To this end, we first train a 2D diffusion prior. Given the camera pose in novel views, our approach transfers the camera information into explicit hand representations, including hand depth and skeleton images. We propose a global hand embedding to control the diffusion model based on these hand representations. We then learn a 3D Gaussian splatting for novel-view rendering using the diffusion prior. However, occluded objects present a persistent challenge. To address this issue, we further introduce local hand embedding, where a contact field is defined in the 3D Gaussian Splatting. We leverage contact information to guide the rendering in the contact field. Extensive experiments on the HO3D and DexYCB datasets demonstrate that our method significantly outperforms state-of-the-art novel-view synthesis for hand-object interactions.

MAIR++: Improving Multi-view Attention Inverse Rendering with Implicit Lighting Representation.

In this paper, we propose a scene-level inverse rendering framework that uses multi-view images to decompose the scene into geometry, SVBRDF, and 3D spatially-varying lighting. While multi-view images have been widely used for object-level inverse rendering, scene-level inverse rendering has primarily been studied using single-view images due to the lack of a dataset containing high dynamic range multi-view images with ground-truth geometry, material, and spatially-varying lighting. To improve the quality of scene-level inverse rendering, a novel framework called Multi-view Attention Inverse Rendering (MAIR) was recently introduced. MAIR performs scene-level multi-view inverse rendering by expanding the OpenRooms dataset, designing efficient pipelines to handle multi-view images, and splitting spatially-varying lighting. Although MAIR showed impressive results, its lighting representation is fixed to spherical Gaussians, which limits its ability to render images realistically. Consequently, MAIR cannot be directly used in applications such as material editing. Moreover, its multi-view aggregation networks have difficulties extracting rich features because they only focus on the mean and variance between multi-view features. In this paper, we propose its extended version, called MAIR++. MAIR++ addresses the aforementioned limitations by introducing an implicit lighting representation that accurately captures the lighting conditions of an image while facilitating realistic rendering. Furthermore, we design a directional attention-based multi-view aggregation network to infer more intricate relationships between views. Experimental results show that MAIR++ not only outperforms MAIR and single-view-based methods but also demonstrates robust performance on unseen real-world scenes.

PAPRec: 3D Point Cloud Reconstruction Based on Prior-Guided Adaptive Probabilistic Network.

Inferring a complete 3D shape from a single-view image is an ill-posed problem. The proposed methods often have problems such as insufficient feature expression, unstable training and limited constraints, resulting in a low accuracy and ambiguity reconstruction. To address these problems, we propose a prior-guided adaptive probabilistic network for single-view 3D reconstruction, called PAPRec. In the training stage, PAPRec encodes a single-view image and its corresponding 3D prior into image feature distribution and point cloud feature distribution, respectively. PAPRec then utilizes a latent normalizing flow to fit the two distributions and obtains a latent vector with rich cues. PAPRec finally introduces an adaptive probabilistic network consisting of a shape normalizing flow and a diffusion model in order to decode the latent vector as a complete 3D point cloud. Unlike the proposed methods, PAPRec fully learns the global and local features of objects by innovatively integrating 3D prior guidance and the adaptive probability network under the optimization of a loss function combining prior, flow and diffusion losses. The experimental results on the public ShapeNet dataset show that PAPRec, on average, improves CD by 2.62%, EMD by 5.99% and F1 by 4.41%, in comparison to several state-of-the-art methods.

Leveraging Single-View Images for Unsupervised 3D Point Cloud Completion

Leveraging Single-View Images for Unsupervised 3D Point Cloud Completion

TEXGen: a Generative Diffusion Model for Mesh Textures

While high-quality texture maps are essential for realistic 3D asset rendering, few studies have explored learning directly in the texture space, especially on large-scale datasets. In this work, we depart from the conventional approach of relying on pre-trained 2D diffusion models for testtime optimization of 3D textures. Instead, we focus on the fundamental problem of learning in the UV texture space itself. For the first time, we train a large diffusion model capable of directly generating high-resolution texture maps in a feed-forward manner. To facilitate efficient learning in high-resolution UV spaces, we propose a scalable network architecture that interleaves convolutions on UV maps with attention layers on point clouds. Leveraging this architectural design, we train a 700 million parameter diffusion model that can generate UV texture maps guided by text prompts and single-view images. Once trained, our model naturally supports various extended applications, including text-guided texture inpainting, sparse-view texture completion, and text-driven texture synthesis. The code is available at https://github.com/CVMI-Lab/TEXGen.

MV2MV: Multi-View Image Translation via View-Consistent Diffusion Models

Image translation has various applications in computer graphics and computer vision, aiming to transfer images from one domain to another. Thanks to the excellent generation capability of diffusion models, recent single-view image translation methods achieve realistic results. However, directly applying diffusion models for multi-view image translation remains challenging for two major obstacles: the need for paired training data and the limited view consistency. To overcome the obstacles, we present a first unified multi-view image to multi-view image translation framework based on diffusion models, called MV2MV. Firstly, we propose a novel self-supervised training strategy that exploits the success of off-the-shelf single-view image translators and the 3D Gaussian Splatting (3DGS) technique to generate pseudo ground truths as supervisory signals, leading to enhanced consistency and fine details. Additionally, we propose a latent multi-view consistency block, which utilizes the latent-3DGS as the underlying 3D representation to facilitate information exchange across multi-view images and inject 3D prior into the diffusion model to enforce consistency. Finally, our approach simultaneously optimizes the diffusion model and 3DGS to achieve a better trade-off between consistency and realism. Extensive experiments across various translation tasks demonstrate that MV2MV outperforms task-specific specialists in both quantitative and qualitative.

Self-supervised reconstruction of re-renderable facial textures from single image

Self-supervised reconstruction of re-renderable facial textures from single image

Enhanced Electroacoustic Tomography with Supervised Learning for Real-time Electroporation Monitoring.

Nanosecond pulsed electric fields (nsPEF)-based electroporation is a new therapy modality potentially synergized with radiation therapy to improve treatment outcomes. To verify its treatment accuracy intraoperatively, electroacoustic tomography (EAT) has been developed to monitor in-vivo electric energy deposition by detecting ultrasound signals generated by nsPEFs in real-time. However, utility of EAT is limited by image distortions due to the limited-angle view of ultrasound transducers. This study proposed a supervised learning-based workflow to address the ill-conditioning in EAT reconstruction. Electroacoustic signals were detected by a linear array and initially reconstructed into EAT images, which were then fed into a deep learning model for distortion correction. In this study, 56 distinct electroacoustic data sets from nsPEFs of different intensities and geometries were collected experimentally, avoiding simulation-to-real-world variations. Forty-six data were used for model training and 10 for testing. The model was trained using supervised learning, enabled by a custom rotating platform to acquire paired full-view and single-view signals for the same electric field. The proposed method considerably improved the image quality of linear array-based EAT, generating pressure maps with accurate and clear structures. Quantitatively, the enhanced single-view images achieved a low-intensity error (RMSE: 0.018), high signal-to-noise ratio (PSNR: 35.15), and high structural similarity (SSIM: 0.942) compared to the reference full-view images. This study represented a pioneering stride in achieving high-quality EAT using a single linear array in an experimental environment, which improves EAT's utility in real-time monitoring for nsPEF-based electroporation therapy.

Structure-Guided Image Inpainting Based on Multi-Scale Attention Pyramid Network

Current single-view image inpainting methods often suffer from low image information utilization and suboptimal repair outcomes. To address these challenges, this paper introduces a novel image inpainting framework that leverages a structure-guided multi-scale attention pyramid network. This network consists of a structural repair network and a multi-scale attention pyramid semantic repair network. The structural repair component utilizes a dual-branch U-Net network for robust structure prediction under strong constraints. The predicted structural view then serves as auxiliary information for the semantic repair network. This latter network exploits the pyramid structure to extract multi-scale features of the image, which are further refined through an attention feature fusion module. Additionally, a separable gated convolution strategy is employed during feature extraction to minimize the impact of invalid information from missing areas, thereby enhancing the restoration quality. Experiments conducted on standard datasets such as Paris Street View and CelebA demonstrate the superiority of our approach over existing methods through quantitative and qualitative comparisons. Further ablation studies, by incrementally integrating proposed mechanisms into a baseline model, substantiate the effectiveness of our multi-view restoration strategy, separable gated convolution, and multi-scale attention feature fusion.

Application of computer vision and machine learning in morphological characterization of Adansonia digitata fruits

Application of computer vision and machine learning in morphological characterization of Adansonia digitata fruits

3D voxel reconstruction from single-view image based on cross-domain feature fusion

3D voxel reconstruction from single-view image based on cross-domain feature fusion

Novel digital-based approach for evaluating wine components’ intake: A deep learning model to determine red wine volume in a glass from single-view images

Novel digital-based approach for evaluating wine components’ intake: A deep learning model to determine red wine volume in a glass from single-view images

3D colored object reconstruction from a single view image through diffusion

3D colored object reconstruction from a single view image through diffusion

Monocular 3D Hand Mesh Recovery via Dual Noise Estimation

Current parametric models have made notable progress in 3D hand pose and shape estimation. However, due to the fixed hand topology and complex hand poses, current models are hard to generate meshes that are aligned with the image well. To tackle this issue, we introduce a dual noise estimation method in this paper. Given a single-view image as input, we first adopt a baseline parametric regressor to obtain the coarse hand meshes. We assume the mesh vertices and their image-plane projections are noisy, and can be associated in a unified probabilistic model. We then learn the distributions of noise to refine mesh vertices and their projections. The refined vertices are further utilized to refine camera parameters in a closed-form manner. Consequently, our method obtains well-aligned and high-quality 3D hand meshes. Extensive experiments on the large-scale Interhand2.6M dataset demonstrate that the proposed method not only improves the performance of its baseline by more than 10% but also achieves state-of-the-art performance. Project page: https://github.com/hanhuili/DNE4Hand.

CDPNet: Cross-Modal Dual Phases Network for Point Cloud Completion

Point cloud completion aims at completing shapes from their partial. Most existing methods utilized shape’s priors information for point cloud completion, such as inputting the partial and getting the complete one through an encoder-decoder deep learning structure. However, it is very often to easily cause the loss of information in the generation process because of the invisibility of missing areas. Unlike most existing methods directly inferring the missing points using shape priors, we address it as a cross-modality task. We propose a new Cross-modal Dual Phases Network (CDPNet) for shape completion. Our key idea is that the global information of the shape is obtained from the extra single-view image, and the partial point clouds provide the geometric information. After that, the multi-modal features jointly guide the specific structural information. To learn the geometric details of the shape, we chose to use patches to preserve the local geometric feature. In this way, we can generate shapes with enough geometric details. Experimental results show that our method achieves state-of-the-art performance on point cloud completion.

Reconstruction of 3D Information of Buildings from Single-View Images Based on Shadow Information

Accurate building geometry information is crucial for urban planning in constrained spaces, fueling the growing demand for large-scale, high-precision 3D city modeling. Traditional methods like oblique photogrammetry and LiDAR prove time consuming and expensive for low-cost 3D reconstruction of expansive urban scenes. Addressing this challenge, our study proposes a novel approach to leveraging single-view remote sensing images. By integrating shadow information with deep learning networks, our method measures building height and employs a semantic segmentation technique for single-image high-rise building reconstruction. In addition, we have designed complex shadow measurement algorithms and building contour correction algorithms to improve the accuracy of building models in conjunction with our previous research. We evaluate the method’s precision, time efficiency, and applicability across various data sources, scenarios, and scales. The results demonstrate the rapid and accurate acquisition of 3D building data with maintained geometric accuracy (mean error below 5 m). This approach offers an economical and effective solution for large-scale urban modeling, bridging the gap in cost-efficient 3D reconstruction techniques.

HieGeneNet: hierarchical implicit 3D shapes generation with single view images

HieGeneNet: hierarchical implicit 3D shapes generation with single view images

Structure-aware multi-view image inpainting using dual consistency attention

Structure-aware multi-view image inpainting using dual consistency attention