Normal Map Research Articles

Enhanced Multi-Scale Attention-Driven 3D Human Reconstruction from Single Image

Due to the inherent limitations of a single viewpoint, reconstructing 3D human meshes from a single image has long been a challenging task. While deep learning networks enable us to approximate the shape of unseen sides, capturing the texture details of the non-visible side remains difficult with just one image. Traditional methods utilize Generative Adversarial Networks (GANs) to predict the normal maps of the non-visible side, thereby inferring detailed textures and wrinkles on the model’s surface. However, we have identified challenges with existing normal prediction networks when dealing with complex scenes, such as a lack of focus on local features and insufficient modeling of spatial relationships.To address these challenges, we introduce EMAR—Enhanced Multi-scale Attention-Driven Single-Image 3D Human Reconstruction. This approach incorporates a novel Enhanced Multi-Scale Attention (EMSA) mechanism, which excels at capturing intricate features and global relationships in complex scenes. EMSA surpasses traditional single-scale attention mechanisms by adaptively adjusting the weights between features, enabling the network to more effectively leverage information across various scales. Furthermore, we have improved the feature fusion method to better integrate representations from different scales. This enhanced feature fusion allows the network to more comprehensively understand both fine details and global structures within the image. Finally, we have designed a hybrid loss function tailored to the introduced attention mechanism and feature fusion method, optimizing the network’s training process and enhancing the quality of reconstruction results. Our network demonstrates significant improvements in performance for 3D human model reconstruction. Experimental results show that our method exhibits greater robustness to challenging poses compared to traditional single-scale approaches.

Accuracy of endocardial abnormal electrogram characterization using a 17-AHA-segmental late boundary and voltage-dependent sensitivity annotation for ventricular tachycardia ablation

Abstract Background Substrate-based catheter ablation of ventricular tachycardia (VT) has become the mainstay therapy for patients with scar-related VT. Correct identification and timing of abnormal near-field EGMs and discerning it from far-field EGMs, is key for procedural success. The novel CARTO™ Late Activation Mapping (LAM) module provides fast and accurate annotation of near-field EGMs including late abnormal ventricular activation. For the algorithm to succeed, a late boundary (LB) annotation line should be positioned immediately behind the normal near-field EGM, however, since the timing of normal local near-field activation varies greatly throughout the endocardium, there is currently no consensus for the optimal LB positioning. We hypothesized that 17-AHA segmental annotation of the endocardial LB followed by scar-dependent sensitivity settings, will improve the accuracy of the near-field EGM characterization. Purpose To evaluate the accuracy of endocardial abnormal EGM characterization using a 17-AHA-segmental late boundary and voltage-dependent sensitivity annotation as compared to conventional mid-QRS LB positioning. Methods Five high-resolution substrate voltage maps were obtained from 5 patients (80% male, 68±7 years old, 80% ischemic cardiomyopathy) that underwent a catheter-based ablation for scar-related VT. A personalized CT-derived anatomical reconstruction of the 17-segment AHA model was co-registered offline with the electroanatomic voltage map, using ADAS3D software. Per segment, individual LB timings were set at the highest sensitivity. A gold standard annotation map was created by two independent experts after manual scrutiny of individual EGMs. Next, the expert map was compared to automatic maps created from segmental LB and mid-QRS LB positions at normal, high, highest or mixed (normal/high or normal/highest) sensitivities, focusing on EGM voltage and activation accuracy. Results A mean of 2041±1028 points per map were analyzed. Correct correlation of EGM annotation was found on average in 91% (activation) and 81% (voltage) for all segmental LB positioned maps (segLB), and in 83% (activation) and 77% (voltage) for all mid-QRS LB positioned maps (mid-QRSLB). The highest correlation with expert map for both activation and voltage annotation was found in the mixed sensitivity normal/highest map with segmental LB positioning (activation 98%, voltage 88%). The correlation on activation was significantly superior in the segLB map when compared to the normal sensitivity maps (p=0.0008 for mid-QRSLB and p=0.02 for segLB). For correlation on voltage, the segLBmap with mixed sensitivity was significantly superior compared to three mid-QRSLB maps at normal (p=0.04), high and highest (p=0.01) sensitivity. Conclusion Individualized, 17-AHA-segmental late boundary and voltage-dependent sensitivity annotation provides a possible accurate and clinically-relevant framework for substrate assessment. Prospective validation is needed.

New Generalized Derivatives for Solving Variational Inequalities Using the Nonsmooth Newton Methods

AbstractVariational inequality (VI) generalizes many mathematical programming problems and has a wide variety of applications. One class of VI solution methods is to reformulate a VI into a normal map nonsmooth equation system, which is then solved using nonsmooth equation-solving techniques. In this article, we propose a first practical approach for furnishing B-subdifferential elements of the normal map, which in turn enables solving the normal map equation system using variants of the B-subdifferential-based nonsmooth Newton method. It is shown that our new method requires less stringent conditions to achieve local convergence than some other established methods, and thus guarantees convergence in certain cases where other methods may fail. We compute a B-subdifferential element using the LD-derivative, which is a recently established generalized derivative concept. In our new approach, an LD-derivative is computed by solving a sequence of strictly convex quadratic programs, which can be terminated early under certain conditions. Numerical examples are provided to illustrate the convergence properties of our new method, based on a proof-of-concept implementation in Python.

Lappan’s five-point theorem for $$\varphi $$-normal harmonic mappings

Lappan’s five-point theorem for $$\varphi $$-normal harmonic mappings

3D Head Pose Estimation via Normal Maps: A Generalized Solution for Depth Image, Point Cloud, and Mesh

Head pose estimation plays a crucial role in various applications, including human–machine interaction, autonomous driving systems, and 3D reconstruction. Current methods address the problem primarily from a 2D perspective, which limits the efficient utilization of 3D information. Herein, a novel approach, called pose orientation‐aware network (POANet), which leverages normal maps for orientation information embedding, providing abundant and robust head pose information, is introduced. POANet incorporates the axial signal perception module and the rotation matrix perception module, these lightweight modules make the approach achieve state‐of‐the‐art (SOTA) performance with few computational costs. This method can directly analyze various topological 3D data without extensive preprocessing. For depth images, POANet outperforms existing methods on the Biwi Kinect head pose dataset, reducing the mean absolute error (MAE) by ≈30% compared to the SOTA methods. POANet is the first method to perform rigid head registration in a landmark‐free manner. It also incorporates few‐shot learning capabilities and achieves an MAE of about on the Headspace dataset. These features make POANet a superior alternative to traditional generalized Procrustes analysis for mesh data processing, offering enhanced convenience for human phenotype studies.

FaSS-MVS: Fast Multi-View Stereo with Surface-Aware Semi-Global Matching from UAV-Borne Monocular Imagery.

With FaSS-MVS, we present a fast, surface-aware semi-global optimization approach for multi-view stereo that allows for rapid depth and normal map estimation from monocular aerial video data captured by unmanned aerial vehicles (UAVs). The data estimated by FaSS-MVS, in turn, facilitate online 3D mapping, meaning that a 3D map of the scene is immediately and incrementally generated as the image data are acquired or being received. FaSS-MVS is composed of a hierarchical processing scheme in which depth and normal data, as well as corresponding confidence scores, are estimated in a coarse-to-fine manner, allowing efficient processing of large scene depths, such as those inherent in oblique images acquired by UAVs flying at low altitudes. The actual depth estimation uses a plane-sweep algorithm for dense multi-image matching to produce depth hypotheses from which the actual depth map is extracted by means of a surface-aware semi-global optimization, reducing the fronto-parallel bias of Semi-Global Matching (SGM). Given the estimated depth map, the pixel-wise surface normal information is then computed by reprojecting the depth map into a point cloud and computing the normal vectors within a confined local neighborhood. In a thorough quantitative and ablative study, we show that the accuracy of the 3D information computed by FaSS-MVS is close to that of state-of-the-art offline multi-view stereo approaches, with the error not even an order of magnitude higher than that of COLMAP. At the same time, however, the average runtime of FaSS-MVS for estimating a single depth and normal map is less than 14% of that of COLMAP, allowing us to perform online and incremental processing of full HD images at 1-2 Hz.

Analysis of myocardial T1, T2, and T2* values by age, sex, and cardiac segments in normal population: a prospective study.

This study examines myocardial T1, T2, and T2* values in a sizable cohort of healthy volunteers, analyzing variations by age, sex, and cardiac segments. It offers a novel approach to defining normal parametric mapping boundaries and represents the first comprehensive study of its kind in Turkey. Our prospective study was conducted between August 2021 and August 2022. Healthy volunteers aged 20-80 were grouped, with at least eight females and eight males per decade. Cardiac MRI examination measured T1 and T2 times in 16 left ventricle segments using parametric mapping techniques on a 1.5 Tesla MRI device. T2* mapping was also performed on the mid-section interventricular septum. The data analysis considered the impact of age, sex, and segments. One hundred eighteen cases were included in the study. Female volunteers observed significantly higher T1, T2, and T2* values than male volunteers. For the T2* and T1 times, significantly lower values were detected in women over 50 than those under 50. It was observed that the Midventricular approach (middle section) gave closer results than the Midventricular Septal approach (septal region of middle section) in predicting Global times. We present the normal reference ranges for cardiac T1, T2, and T2* times in a large cohort of healthy volunteers with homogeneous sex and age distribution. Sex was the most influential factor in our study. Therefore, we suggest using separate reference values for males, and females above and below 50 years old, instead of the standard reference intervals that do not account for specified sex in current guidelines.

A quantitative approach to reflectance transformation imaging in profilometric applications

Reflectance transformation imaging (RTI) is a technique employed to assess both the intensity and directional properties of light reflected from an object, aiming to visualize an object under different incident light directions. This technique has quickly become a commonly used approach for the documentation, acquisition and deciphering for cultural heritage objects, because it enables to enhance and highlight image details. The output of this mathematical synthesis technique, which elaborates illumination information, is contained in a pseudocolour image called normal map. Starting from this image, this technique allows to obtain the reconstruction of the outline of semi-flat objects by integrating the normal map given by the RTI, i.e. normal integration. The technique of normal integration has been known for many years, being widely used in graphical modelling across various fields of 3D imaging. In the context of cultural heritage, this particular form of 3D modelling has already found application in the reconstruction of semi-flat objects, albeit with some limitations in the accurate portraying of low spatial frequencies. However, quantitative and systematic studies on the efficiency of 3D rendering of objects using RTI and normal integration have not yet been conducted. In the context of this research, the quantitative evaluation of the accuracy of the RTI technique with normal integration for profilometric applications on semi-flat objects is proposed and a study on its fidelity in reproducing the considered artefact is conducted. For this purpose, ad hoc profilometric targets were designed, realized and analysed to quantitatively compare the results of RTI and normal integration with different standardized techniques: micro-photogrammetry, laser scanning and optical profilometry.Graphical abstract

Pose-Driven Compression for Dynamic 3D Human via Human Prior Models.

To cost-effectively transmit high-quality dynamic 3D human images in immersive multimedia applications, efficient data compression is crucial. Unlike existing methods that focus on reducing signal-level reconstruction errors, we propose the first dynamic 3D human compression framework based on human priors. The layered coding architecture significantly enhances the perceptual quality while also supporting a variety of downstream tasks, including visual analysis and content editing. Specifically, a high-fidelity pose-driven Avatar is generated from the original frames as the basic structure layer to implicitly represent the human shape. Then, human movements between frames are parameterized via a commonly-used human prior model, i.e., the Skinned Multi-Person Linear Model (SMPL), to form the motion layer and drive the Avatar. Furthermore, the normals are also introduced as an enhancement layer to preserve fine-grained geometric details. Finally, the Avatar, SMPL parameters, and normal maps are efficiently compressed into layered semantic bitstreams. Extensive qualitative and quantitative experiments show that the proposed framework remarkably outperforms other state-of-the-art 3D codecs in terms of subjective quality with only a few bits. More notably, as the size or frame number of the 3D human sequence increases, the superiority of our framework in perceptual quality becomes more significant while saving more bitrates.

Coplane-constrained sparse depth sampling and local depth propagation for depth estimation

Depth estimation with sparse reference has emerged recently, and predicts depth map from a monocular image and a set of depth reference samples. Previous works randomly select reference samples by sensors, leading to severe depth bias as this sampling is independent to image semantic and neglects the unbalance of depth distribution in regions. This paper proposes a Coplane-Constrained sparse Depth (CCD) sampling to explore representative reference samples, and design a Local Depth Propagation (LDP) network for complete the sparse point cloud map. This can capture diverse depth information and diffuse the valid points to neighbors with geometry prior. Specifically, we first construct the surface normal map and detect coplane pixels by superpixel segmenting for sampling references, whose depth can be represented by that of superpixel centroid. Then, we introduce local depth propagation to obtain coarse-level depth map with geometric information, which dynamically diffuses the depth from the reference to neighbors based on local planar assumption. Further, we generate the fine-level depth map by devising a pixel-wise focal loss, which imposes the semantic and geometry calibration on pixels with low confidence in coarse-level prediction. Extensive experiments on public datasets demonstrate that our model outperforms SOTA depth estimation and completion methods.

Modeling Realistic Clothing From a Single Image Under Normal Guide.

We propose a robust and highly realistic clothing modeling method to generate a 3D clothing model with visually consistent clothing style and wrinkles distribution from a single RGB image. Notably, this entire process only takes a few seconds. Our high-quality clothing results benefit from the idea of combining learning and optimization, making it highly robust. First, we use the neural networks to predict the normal map, a clothing mask, and a learning-based clothing model from input images. The predicted normal map can effectively capture high-frequency clothing deformation from image observations. Then, by introducing a normal-guided clothing fitting optimization, the normal maps are used to guide the clothing model to generate realistic wrinkles details. Finally, we utilize a clothing collar adjustment strategy to stylize clothing results using predicted clothing masks. An extended multi-view version of the clothing fitting is naturally developed, which can further improve the realism of the clothing without tedious effort. Extensive experiments have proven that our method achieves state-of-the-art clothing geometric accuracy and visual realism. More importantly, it is highly adaptable and robust to in-the-wild images. Further, our method can be easily extended to multi-view inputs to improve realism. In summary, our method can provide a low-cost and user-friendly solution to achieve realistic clothing modeling.

Monocular Depth and Ego-motion Estimation with Scale Based on Superpixel and Normal Constraints

3D perception in intelligent VR/AR and autonomous vehicles (AV) applications is critical and attracting significant attention. The self-supervised monocular depth and ego-motion estimation serves as a more intelligent learning approach that provides the required scene depth and location for 3D perception. However, the existing self-supervised learning methods suffer from scale ambiguity, boundary blur and imbalanced depth distribution, limiting the practical applications of VR/AR and AV. In this paper, we propose a new self-supervised learning framework based on superpixel and normal constraints to address these problems. Specifically, we formulate a novel 3D edge structure consistency loss to alleviate the boundary blur of depth estimation. To address the scale ambiguity of estimated depth and ego-motion, we propose a novel surface normal network for efficient camera height estimation. The surface normal network is composed of a deep fusion module and a full-scale hierarchical feature aggregation module. Meanwhile, to realize the global smoothing and boundary discriminability of the predicted normal map, we introduce a novel fusion loss which is based on the consistency constraints of the normal in edge domains and superpixel regions. Experiments are conducted on several benchmarks, and the results illustrate that the proposed approach outperforms the state-of-the-art methods in depth, ego-motion and surface normal estimation.

Interpretable deep generative model in myocardial perfusion SPECT images based on the creation of patient-matched normal maps

Interpretable deep generative model in myocardial perfusion SPECT images based on the creation of patient-matched normal maps

Automated quantitative assessment of bone contusions and overlying articular cartilage following anterior cruciate ligament injury.

Quantitative methods to characterize bone contusions and associated cartilage injury remain limited. We combined standardized voxelwise normalization and 3D mapping to automate bone contusion segmentation post-anterior cruciate ligament (ACL) injury and evaluate anomalies in articular cartilage overlying bone contusions. Forty-five patients (54% female, 26.4 ± 11.8 days post-injury) with an ACL tear underwent 3T magnetic resonance imaging of their involved and uninvolved knees. A novel method for voxelwise normalization and 3D anatomical mapping was used to automate segmentation, labeling, and localization of bone contusions in the involved knee. The same mapping system was used to identify the associated articular cartilage overlying bone lesions. Mean regional T1ρ was extracted from articular cartilage regions in both the involved and uninvolved knees for quantitative paired analysis against ipsilateral cartilage within the same compartment outside of the localized bone contusion. At least one bone contusion lesion was detected in the involved knee within the femur and/or tibia following ACL injury in 42 participants. Elevated T1ρ (p = 0.033) signal were documented within the articular cartilage overlying the bone contusions resulting from ACL injury. In contrast, the same cartilaginous regions deprojected onto the uninvolved knees showed no ipsilateral differences (p = 0.795). Automated bone contusion segmentation using standardized voxelwise normalization and 3D mapping deprojection identified altered cartilage overlying bone contusions in the setting of knee ACL injury.

Review of Application YOLOv8 in Medical Imaging

Deep learning has revolutionized medical imaging analysis, with YOLOv8 emerging as a promising tool forvarious tasks like lesion detection, organ segmentation and disease classification. This review investigates YOLOv8’sapplications across diverse medical imaging modalities (X-Ray, CT-Scan and MRI). We conducted a systematic literaturesearch across databases like Pubmed, ScienceDirect and IEEE to identify relevant studies evaluating YOLOv8’sperformance in medical imaging analysis. YOLOv8 achieved high performance for meningioma and pituitary tumorswith and without data augmentation (precision >0.92, recall >0.90, mAP >0.93). Glioma detection showed lowerperformance but still promising results (precision >0.86, recall >0.81, mAP >0.86). Breast cancer detection with SGDoptimizer yielded best performance with an average mAP of 0.87 for mass detection. The model achieved high accuracyin detecting normal (mAP 0.939) and malignant lesions (mAP 0.911). YOLO v8 on Dental radiograph successfullydetected cavities, impacted teeth, fillings and implants (precision of >0.82, recall of >0.78 and F1-Score of >0.80). Lastly,for lung disease classification, YOLOv8 achieved high accuracy (99.8% training and 90% validation) in classifyingnormal, COVID-19, influenza and lung cancer disease. With the importance to improve clinical decision-making andpatient outcomes in healthcare, the YOLOv8 algorthm underscores the importance of pre-processing, augmentation andoptimization of key hyperparameters.

A combined approach of SFM-MVS photogrammetry and reflectance transformation imaging to enhance 3D reconstructions

One of the problems that arises when applying 3D Structure from Motion and Multi-View-Stereo photogrammetry to certain heritage objects characterized by minimal surface features and monochrome textures is that the obtained 3D models have low quality and accuracy. Within such scenarios, the smallest lengths that can be measured on it with high accuracy is usually one or more orders of magnitude worse than the maximum resolution of the input photographs. Additionally, when contrasted with outputs from scanning systems, these reconstructions tend to exhibit relatively lower quality. This article presents a simple approach wherein the incorporation of normal maps acquired through Reflectance Transformation Imaging technique improves the quality and accuracy of 3D photogrammetric reconstructions (mesh + texture). This enhancement also surpasses the capabilities of some commercial structured light scanning systems and reaches values that are in the order of magnitude of the maximum resolution of the photographs used.

Implicit 3D Human Reconstruction Guided by Parametric Models and Normal Maps.

Accurate and robust 3D human modeling from a single image presents significant challenges. Existing methods have shown potential, but they often fail to generate reconstructions that match the level of detail in the input image. These methods particularly struggle with loose clothing. They typically employ parameterized human models to constrain the reconstruction process, ensuring the results do not deviate too far from the model and produce anomalies. However, this also limits the recovery of loose clothing. To address this issue, we propose an end-to-end method called IHRPN for reconstructing clothed humans from a single 2D human image. This method includes a feature extraction module for semantic extraction of image features. We propose an image semantic feature extraction aimed at achieving pixel model space consistency and enhancing the robustness of loose clothing. We extract features from the input image to infer and recover the SMPL-X mesh, and then combine it with a normal map to guide the implicit function to reconstruct the complete clothed human. Unlike traditional methods, we use local features for implicit surface regression. Our experimental results show that our IHRPN method performs excellently on the CAPE and AGORA datasets, achieving good performance, and the reconstruction of loose clothing is noticeably more accurate and robust.

Simulacra and historical fidelity in digital recreation of lost cultural heritage: Reconstituting period materialities for the period eye

The advancement of digital technologies in art history has opened avenues for reconstructing lost or damaged cultural heritage, a need highlighted by the deteriorated state of many artworks from the 1785 Salon. Grounded in the concept of the “Period Eye” by art historian Michael Baxandall, which emphasizes understanding artworks within their original historical and cultural contexts, this study proposes a subfield focused on Reconstituting Period Materialities for the Period Eye. This methodology bridges comprehensive historical research with generative visual artificial intelligence (AI) technologies, facilitating the creation and immersive virtual reality viewing of artworks. Beyond mere visual replication, the approach aims to recreate the material and textural realities of the period, thereby enabling contemporary audiences to experience these works as they were originally perceived. The process includes replicating building materials using Quixel Megascans, employing AI for generating images of lost artworks, and utilizing normal maps for simulating painting textures, all contributing to an authentic reconstruction of the Salon’s ambiance and materiality. This approach, met with some skepticism from traditional historians and archeologists, asserts that such digital reconstitution, backed by rigorous empirical research and detailed period-specific datasets, yields reconstructions of greater historical accuracy and contextual richness. This mirrors strides in sound archeology, endorsing a similar empirical approach in visual material recreation. The significance of this study is underscored by its potential to enrich our comprehension of historical artworks through a “Period Eye,” blending historical insights with modern technological innovation for a deeper understanding and appreciation of cultural heritage.

Enhancing flood verification using Signal Detection Theory (SDT) and IoT Sensors: A spatial scale evaluation

Inundation mapping by flood simulation is subject to uncertainties which spatial accuracy assessment could not be processed as normal mapping does. This paper presents a comprehensive study on the spatial accuracy of flood verification using Internet of Things (IoT) sensors in several flood-prone regions in Taiwan, including Changhua, Yunlin, Chiayi, Tainan, Kaohsiung, and Pingtung, during three storm events in 2021. The study adopts Signal Detection Theory (SDT) and proposes an ensemble-based approach that combines multiple evaluation metrics to enhance the accuracy and reliability of flood verification in these regions. The study examines the inundation availabilities between flood observation from IoT sensors and flood simulation. A range of spatial scales varies from 100 m to 1000 m and also administrative divisions are considered including neighborhood-level and village-level scales. The results show that increasing the spatial scale generally leads to higher True Positives, indicating improved flood detection, but it also results in increased False Positives, indicating more misclassification of non-flooded points. The findings reveal that the optimal spatial scale for all cities/counties is 450 m which indicates the averaged spatial accuracy of the flood simulation. However, there is notable variation in the optimal spatial scale across different regions. The ensemble approach demonstrates high consistency in determining the optimal spatial scale, as eight out of ten metrics consistently identify the same value. Additionally, the study compares flood verification across administrative divisions in Taiwan, finding that the village-level spatial scale performs best in Changhua and Yunlin counties, while the neighborhood-level scale is preferred in Tainan, Kaohsiung, and Pingtung. This research provides a valuable framework to estimate inundation mapping accuracy under flood verification using IoT sensors in the specified region of Taiwan and highlights the importance of considering local conditions and administrative divisions in determining the optimal spatial scale. These findings can contribute to the improvement of flood detection and monitoring systems in the area with IoT flood sensors.

Ergodic quantum processes on finite von Neumann algebras

Let (M,τ) be a tracial von Neumann algebra with a separable predual and let (Ω,P) be a probability space. A bounded positive random linear operator on L1(M,τ) is a map γ:Ω×L1(M,τ)→L1(M,τ) so that τ(γω(x)a) is measurable for all x∈L1(M,τ) and a∈M, and x↦γω(x) is bounded, positive, and linear almost surely. Given an ergodic T∈Aut(Ω,P), we study quantum processes of the form γTnω∘γTn−1ω∘⋯∘γTmω for m,n∈Z. Using the Hennion metric introduced in [21], we show that under reasonable assumptions such processes collapse to replacement channels exponentially fast almost surely. Of particular interest is the case when γω is the predual of a normal positive linear map on M. As an example application, we study the clustering properties of normal states that are generated by such random linear operators. These results offer an infinite dimensional generalization of the theorems in [21].