Normal Map Research Articles

Joint-Confidence-guided Multi-Task Learning for 3D Reconstruction and Understanding from Monocular Camera.

3D reconstruction and understanding from monocular camera is a key issue in computer vision. Recent learning-based approaches, especially multi-task learning, significantly achieve the performance of the related tasks. However a few works still have limitation in drawing loss-spatial-aware information. In this paper, we propose a novel Joint-confidence-guided network (JCNet) to simultaneously predict depth, semantic labels, surface normal, and joint confidence map for corresponding loss functions. In details, we design a Joint Confidence Fusion and Refinement (JCFR) module to achieve multi-task feature fusion in the unified independent space, which can also absorb the geometric-semantic structure feature in the joint confidence map. We use confidence-guided uncertainty generated by the joint confidence map to supervise the multi-task prediction across the spatial and channel dimensions. To alleviate the training attention imbalance among different loss functions or spatial regions, the Stochastic Trust Mechanism (STM) is designed to stochastically modify the elements of joint confidence map in the training phase. Finally, we design a calibrating operation to alternately optimize the joint confidence branch and the other parts of JCNet to avoid overfiting. The proposed methods achieve state-of-the-art performance in both geometric-semantic prediction and uncertainty estimation on NYU-Depth V2 and Cityscapes.

Human Bas-Relief Generation From a Single Photograph

We present a semi-automatic method for producing human bas-relief from a single photograph. Given an input photo of one or multiple persons, our method first estimates a 3D skeleton for each person in the image. SMPL models are then fitted to the 3D skeletons to generate a 3D guide model. To align the 3D guide model with the image, we compute a 2D warping field to non-rigidly register the projected contours of the guide model with the body contours in the image. Then the normal map of the 3D guide model is warped by the 2D deformation field to reconstruct an overall base shape. Finally, the base shape is integrated with a fine-scale normal map to produce the final bas-relief. To tackle the complex intra- and inter-body interactions, we design an occlusion relationship resolution method that operates at the level of 3D skeletons with minimal user inputs. To tightly register the model contours to the image contours, we propose a non-rigid point matching algorithm harnessing user-specified sparse correspondences. Experiments demonstrate that our human bas-relief generation method is capable of producing perceptually realistic results on various single-person and multi-person images, on which the state-of-the-art depth and pose estimation methods often fail.

Determination of the activation regions of three deep learning models in infrared images of glaucoma

AbstractPurpose: The main objective of this study is to characterize the activation regions of three Deep Learning models using infrared images of the optic nerve of glaucoma patients.Methods: We have retrospectively collected a sample of patients with primary and secondary open‐angle glaucoma and normal patients. The images in infrared were recorded with a spectral domain optical coherence tomography. Three previously trained models were used, VGG19, ResNet101 and the Shufflenet. Sensitivity, specificity, diagnostic accuracy in training and testing, and ROC area were calculated for all three models. The gradient‐weighted class activation map (GradCAM) was used to obtain the activation regions that highlight the most important components in the images for model prediction. The correlation coefficient between the glaucoma activation maps and between normal activation maps of the three models was calculated, and the location of the activation regions in the glaucoma and normal images was determined for each model.Results: 639 eyes of 415 patients were collected, 432 glaucomatous and 207 normal. The accuracy of the models on the test set was 96.9% (VGG19), 95.3% (ResNet101), and 93.8% (Shufflenet). The activation maps of the ResNet101 and Shufflenet models obtained a high correlation in glaucoma (0.75) and normal (0.68) cases. For the three models, the region of interest was located mainly in the inferior temporal quadrant in 65.2% of the cases (VGG19), in 52.9% (Shufflenet) and in 54.1% (ResNet101).Conclusions: In glaucoma eyes, the regions of interest assessed with the gradient‐weighted class activation maps of the three models analysed are located in more than half of the cases in the inferior temporal quadrant of the optic nerve.

Normal Assisted Pixel-Visibility Learning With Cost Aggregation for Multiview Stereo

Multiple-View Stereo (MVS) aims to reconstruct the dense 3D representations of scenes. MVS has potential applications in the fields of autonomous driving (unstructured environment construction) and robotic navigation (visual-inertial navigation). To mitigate the error of depth estimation in low-textured or occluded regions, this work proposes a two-stage multi-view stereo network for fast and accurate depth estimation. The improvements of this work over the state of the art are as follows: 1) Sparse costs are constructed to jointly predict the initial depth map and surface normal by cost regularization, which proves that the surface normals can be estimated in this way with low memory consumption. 2) A new edge refinement block is developed to refine the coarse surface normal to obtain a fine-grained surface normal map. 3) Instead of using the general variance-based metric to equally aggregate cost, a new content-adaptive cost aggregation mechanism based on the similarity of the neighboring surface normal is designed for reliable cost aggregation. To the best of our knowledge, the proposed work is the first trainable network that leverages surface normal as guidance to capture neighboring pixel-visibility, which is an effective supplement to existing depth/normal estimation frameworks. Experimental results indicate that our method can not only achieve accurate depth estimation for scene perception but also make no concession to the real-time performance and limited memory bottleblock. Multiple-view stereo (MVS) aims to reconstruct the dense 3D representations of scenes. It is widely used in the fields of industrial measurement, autonomous driving, and robotic navigation. To mitigate the error of depth estimation in challenging scenarios, this work proposes a two-stage multi-view stereo network for fast and accurate depth estimation. Our method is the first trainable network that leverages surface normal as pixel-visibility guidance to aggregate reliable cost, which could achieve accurate depth estimation and provide the perception ability for the robot. The proposed method has great potential in the fields of 3D reconstruction, industrial measurement, and robotic navigation to estimate real-time and accurate depth with limited memory consumption.

CAD2Sketch

Concept sketches are ubiquitous in industrial design, as they allow designers to quickly depict imaginary 3D objects. To construct their sketches with accurate perspective, designers rely on longstanding drawing techniques, including the use of auxiliary construction lines to identify midpoints of perspective planes, to align points vertically and horizontally, and to project planar curves from one perspective plane to another. We present a method to synthesize such construction lines from CAD sequences. Importantly, our method balances the presence of construction lines with overall clutter, such that the resulting sketch is both well-constructed and readable, as professional designers are trained to do. In addition to generating sketches that are visually similar to real ones, we apply our method to synthesize a large quantity of paired sketches and normal maps, and show that the resulting dataset can be used to train a neural network to infer normals from concept sketches. 1

Learning Normal Maps for Robust 3D Face Recognition from Kinect Data

Face recognition using 3D scans can be achieved by many approaches, but most of these approaches are based on high quality depth sensors. In this paper, the authors use the normal maps obtained from the Kinect sensor to investigate the usefulness of data augmentation and signal-level fusion derived from depth data captured by a low quality sensor. In this face recognition process, the authors first preprocess the captured 3D scan of each person by cropping the face and reducing the noise; normals are computed and separated into three maps: Nx, Ny, and Nz. the authors combine the three normal maps to form an RGB image; these images are used to train a convolutional neural network. The authors investigate the order of components that yields to the best accuracy and compare it with previous results obtained on CurtinFaces and KinectFaceDB databases, achieving rank one identification rate of 94.04% and 91.35%, respectively.

INV-Flow2PoseNet: Light-Resistant Rigid Object Pose from Optical Flow of RGB-D Images Using Images, Normals and Vertices.

This paper presents a novel architecture for simultaneous estimation of highly accurate optical flows and rigid scene transformations for difficult scenarios where the brightness assumption is violated by strong shading changes. In the case of rotating objects or moving light sources, such as those encountered for driving cars in the dark, the scene appearance often changes significantly from one view to the next. Unfortunately, standard methods for calculating optical flows or poses are based on the expectation that the appearance of features in the scene remains constant between views. These methods may fail frequently in the investigated cases. The presented method fuses texture and geometry information by combining image, vertex and normal data to compute an illumination-invariant optical flow. By using a coarse-to-fine strategy, globally anchored optical flows are learned, reducing the impact of erroneous shading-based pseudo-correspondences. Based on the learned optical flows, a second architecture is proposed that predicts robust rigid transformations from the warped vertex and normal maps. Particular attention is paid to situations with strong rotations, which often cause such shading changes. Therefore, a 3-step procedure is proposed that profitably exploits correlations between the normals and vertices. The method has been evaluated on a newly created dataset containing both synthetic and real data with strong rotations and shading effects. These data represent the typical use case in 3D reconstruction, where the object often rotates in large steps between the partial reconstructions. Additionally, we apply the method to the well-known Kitti Odometry dataset. Even if, due to fulfillment of the brightness assumption, this is not the typical use case of the method, the applicability to standard situations and the relation to other methods is therefore established.

Croatian high-resolution monthly gridded dataset of homogenised surface air temperature

Homogenised climatological series and gridded data are the basis for climate monitoring and climate change detection. Considering this, monthly mean temperatures from 122 Croatian stations were homogenised, and high-resolution monthly gridded data were developed for the 1981–2018 period. Homogenisation needs to be performed on stations from the same climate region; therefore, hierarchical clustering is introduced to define those climate regions in Croatia. The breaks of homogeneity were detected by the standard normal homogeneity test on 54 stations. Regression kriging was applied to produce monthly grids for each month in the analysed period. The quality of the interpolation assessed by leave-one-out cross-validation resulted in a root mean square error of 0.7 °C. The quality of spatial interpolation is supplemented with normalised error maps. The derived homogenised station data and monthly grids are necessary for national climate monitoring, the production of climate normals and the estimation of trends. After 1999, average annual anomalies from the 30-year climate standard normal 1981–2010 were positive and up to 1.4 °C warmer than the average and only occasionally negative. The measured amount, sign and significance of the trend were accurately captured on the trend maps calculated from the monthly maps. Significant strong warming was observed and mapped over the entire Croatian territory in April, June, July, August and November. It was stronger inland than on the coast. Annual trends were significant and ranged from 0.3 °C/decade to 0.7 °C/decade. There was no observational evidence of enhanced elevation-dependent warming over elevations from 750 to 1594 m.

Photometric-Stereo-Based Defect Detection System for Metal Parts.

Automated inspection technology based on computer vision is now widely used in the manufacturing industry with high speed and accuracy. However, metal parts always appear in high gloss or shadow on the surface, resulting in the overexposure of the captured images. It is necessary to adjust the light direction and view to keep defects out of overexposure and shadow areas. However, it is too tedious to adjust the position of the light direction and view the variety of parts' geometries. To address this problem, we design a photometric-stereo-based defect detection system (PSBDDS), which combines the photometric stereo with defect detection to eliminate the interference of highlights and shadows. Based on the PSBDDS, we introduce a photometric-stereo-based defect detection framework, which takes images captured in multiple directional lights as input and obtains the normal map through the photometric stereo model. Then, the detection model uses the normal map as input to locate and classify defects. Existing learning-based photometric stereo methods and defect detection methods have achieved good performance in their respective fields. However, photometric stereo datasets and defect detection datasets are not sufficient for training and testing photometric-stereo-based defect detection methods, thus we create a photometric stereo defect detection (PSDD) dataset using our PSBDDS to eliminate gaps between learning-based photometric stereo and defect detection methods. Furthermore, experimental results prove the effectiveness of the proposed PSBBD and PSDD dataset.

Novel View Synthesis Of Transparent Object From a Single Image

AbstractWe propose a method for converting a single image of a transparent object into multi‐view photo that enables users observing the object from multiple new angles, without inputting any 3D shape. The complex light paths formed by refraction and reflection makes it challenging to compute the lighting effects of transparent objects from a new angle. We construct an encoder–decoder network for normal reconstruction and texture extraction, which enables synthesizing novel views of transparent object from a set of new views and new environment maps using only one RGB image. By simultaneously considering the optical transmission and perspective variation, our network learns the characteristics of optical transmission and the change of perspective as guidance to the conversion from RGB colours to surface normals. A texture extraction subnetwork is proposed to alleviate the contour loss phenomenon during normal map generation. We test our method using 3D objects within and without our training data, including real 3D objects that exists in our lab, and completely new environment maps that we take using our phones. The results show that our method performs better on view synthesis of transparent objects in complex scenes using only a single‐view image.

Fast rendering and display of light field images with a controllable lighting mechanism.

A fast light field (LF) image rendering method with controllable lighting mechanism is proposed and demonstrated. It solves the issue that previous image-based methods could not render and edit lighting effects for LF images. In contrast to previous methods, light cones and normal maps are defined and used to expand the RGBD images into RGBDNθ data, which gives more degrees of freedom to render LF images. Conjugate cameras are used to capture the RGBDN data, which simultaneously solve the pseudoscopic imaging problem. Perspective coherence is used to accelerate the RGBDNθ-based LF rendering process, which has been shown to be on average 30 times faster than the traditional per-viewpoint rendering (PVR) method. Vivid three-dimensional (3D) images with Lambertian reflection and non-Lambertian reflection effects including specular lighting and compound lighting have been reconstructed in 3D space using a homemade LF display system. The proposed method injects more flexibility into the rendering of LF images and can also be used in holographic display, augmented reality, virtual reality, and other fields.

Extended Newton-type Method for Nonsmooth Generalized Equation under n , α -point-based Approximation

Let X and Y be Banach spaces and Ω ⊆ X . Let f : Ω ⟶ Y be a single valued function which is nonsmooth. Suppose that F : X⇉ 2 Y is a set-valued mapping which has closed graph. In the present paper, we study the extended Newton-type method for solving the nonsmooth generalized equation 0 ∈ f x + F x and analyze its semilocal and local convergence under the conditions that f + F − 1 is Lipschitz-like and f admits a certain type of approximation which generalizes the concept of point-based approximation so-called n , α -point-based approximation. Applications of n , α -point-based approximation are provided for smooth functions in the cases n = 1 and n = 2 as well as for normal maps. In particular, when 0 < α < 1 and the derivative of f , denoted ∇ f , is ℓ , α -Hölder continuous, we have shown that f admits 1 , α -point-based approximation for n = 1 while f admits 2 , α -point-based approximation for n = 2 , when 0 < α < 1 and the second derivative of f , denoted ∇ 2 f , is K , α -Hölder. Moreover, we have constructed an n , α -point-based approximation for the normal maps f C + F when f has an n , α -point-based approximation. Finally, a numerical experiment is provided to validate the theoretical result of this study.

Histology-verified myocardial fibrosis and quantification in severe AS patients: correlation with non-invasive LV myocardial tissue assessment

Abstract Background Myocardial fibrosis (MF) is a common finding and a potential adverse prognostic marker in several cardiac diseases, including in severe aortic stenosis (AS). While histological analysis obtained through endomyocardial biopsy remains the gold-standard for MF assessment, non-invasive cardiac imaging may offer surrogate biomarkers for fibrosis. We tried to assess the correlation between MF quantification at histopathology and cardiac magnetic resonance (CMR)-derived tissue characterization data in patients with severe AS. Methodology Single-center prospective cohort enrolling 71 patients with severe symptomatic high-gradient AS undergoing surgical aortic valve replacement (SAVR) (mean age 71±9 years; 49% male, mean valvular transaortic gradient 60±20 mmHg; mean left ventricle [LV] ejection fraction 58±9%). Those with past history of myocardial infarction or cardiomyopathy were excluded. All patients underwent pre-operative CMR study with LV tissue characterization and quantification. Normal T1 mapping value was defined as >1021ms as per center protocol. Myocardial tissue was obtained during SAVR either through myocardial biopsy at basal LV septum or harvested from surgical myectomy specimens. Masson's trichrome stain was used for collagen/fibrosis assessment. Automatic quantification was obtained at QuPathTM digital pathology software after applying a dedicated artificial intelligence algorithm on ultra-high-resolution digital slide scanning images. Results Histology-confirmed MF was observed in all patients (median percentage of fibrotic myocardial tissue 15% [IQR 9–22%]). Median global T1 mapping and extracellular volume (ECV) percentage was 1048ms (IQR 1027–1078) and 24% (IQR 20–30%), respectively. Late gadolinium enhancement (LGE) with a non-ischemic pattern was present in 42 patients (59%) with a median LGE mass of 5.8g [IQR 1.0–10.2]; median percentage of 3.7% [IQR 0.6–10.4]. While neither T1 mapping (global or basal LV septum), ECV nor LGE had any significant correlation with histology-confirmed MF (Figure 1) the vast majority had significantly elevated global and basal LV septum T1 mapping – 81% and 92%, respectively. Conclusion In this single-center prospective study, microscopic MF was present in all patients with severe symptomatic high-gradient AS, was accompanied by elevated T1 mapping values but no correlation was found between myocardial fibrosis at histopathology analysis and CMR-derived LV tissue characterization parameters. This may not only stem from sampling (single point biopsy vs. whole myocardial tissue assessment) but also from distinct evaluation of different types of fibrosis by different methods. Funding Acknowledgement Type of funding sources: None.

OHM: GPU Based Occupancy Map Generation

Occupancy grid maps (OGMs) are fundamental to most systems for autonomous robotic navigation. However, CPU-based implementations struggle to keep up with data rates from modern 3D lidar sensors, and provide little capacity for modern extensions which maintain richer voxel representations. This article presents Occupancy Homogenous Mapping (OHM), our open source, GPU-based OGM framework. We show how the algorithms can be mapped to GPU resources, resolving difficulties with contention to obtain a successful implementation. The implementation supports many modern OGM algorithms including Normal Distributions Transform-Occupancy Maps (NDT-OM), Normal Distributions Transform-Traversability Maps (NDT-TM), decay-rate and Truncated Sign Distance Function (TSDF). A thorough performance evaluation is presented based on tracked and quadruped Uncrewed Ground Vehicle (UGV) platforms and UAVs, and data sets from both outdoor and subterranean environments. The results demonstrate excellent performance improvements both offline, and for online processing in embedded platforms. Finally, we describe how OHM was a key enabler for the UGV navigation solution for our entry in the Defense Advanced Research Projects Agency (DARPA) Subterranean Challenge, which placed second at the Final Event.

Color‐mapped noise vector fields for generating procedural micro‐patterns

AbstractStochastic micro‐patterns successfully enhance the realism of virtual scenes. Procedural models using noise combined with transfer functions are extremely efficient. However, most patterns produced today employ 1D transfer functions, which assign color, transparency, or other material attributes, based solely on the single scalar quantity of noise. Multi‐dimensional transfer functions have received widespread attention in other fields, such as scientific volume rendering. But their potential has not yet been well explored for modeling micro‐patterns in the field of procedural texturing. We propose a new procedural model for stochastic patterns, defined as the composition of a bi‐dimensional transfer function (a.k.a. color‐map) with a stochastic vector field. Our model is versatile, as it encompasses several existing procedural noises, including Gaussian noise and phasor noise. It also generates a much larger gamut of patterns, including locally structured patterns which are notoriously difficult to reproduce. We leverage the Gaussian assumption and a tiling and blending algorithm to provide real‐time generation and filtering. A key contribution is a real‐time approximation of the second order statistics over an arbitrary pixel footprint, which enables, in addition, the filtering of procedural normal maps. We exhibit a wide variety of results, including Gaussian patterns, profiled waves, concentric and non‐concentric patterns.

Certain properties of normal meromorphic and normal harmonic mappings

Certain properties of normal meromorphic and normal harmonic mappings

SpongeCake: A Layered Microflake Surface Appearance Model

In this article, we propose SpongeCake: A layered BSDF model where each layer is a volumetric scattering medium, defined using microflake or other phase functions. We omit any reflecting and refracting interfaces between the layers. The first advantage of this formulation is that an exact and analytic solution for single scattering, regardless of the number of volumetric layers, can be derived. We propose to approximate multiple scattering by an additional single-scattering lobe with modified parameters and a Lambertian lobe. We use a parameter mapping neural network to find the parameters of the newly added lobes to closely approximate the multiple scattering effect. Despite the absence of layer interfaces, we demonstrate that many common material effects can be achieved with layers of SGGX microflake and other volumes with appropriate parameters. A normal mapping effect can also be achieved through mapping of microflake orientations, which avoids artifacts common in standard normal maps. Thanks to the analytical formulation, our model is very fast to evaluate and sample. Through various parameter settings, our model is able to handle many types of materials, like plastics, wood, cloth, and so on, opening a number of practical applications.

Role of Cardiac MRI Including LGE, T1 and T2 Mapping in the Assessment of Cardiac Involvement in Patients of Nonspecific Aorto-arteritis: A Prospective Study.

Objective Nonspecific aorto-arteritis (NSAA) may involve the myocardium in the form of edema and fibrosis. We conducted this study to investigate role of cardiac MRI including late gadolinium enhancement (LGE), T1 and T2 mapping in the assessment of cardiac involvement in NSAA. Methods and Materials Over the period between 2016 and 2019, 36 patients with NSAA presenting with uncontrolled hypertension, left ventricular dysfunction, congestive cardiac failure, or tachyarrhythmia were included in the study. We also had 16 voluntary control patients for providing normal T1 and T2 mapping values. Results The average age of patients was 27.1 years and the majority were females. MRI is more sensitive than echocardiography in the detection of LV dysfunction and RWMA. Out of 36 patients, 10 (27.8%) had LGE. The most common pattern of midmyocardial enhancement was present in 5 out of 10 patients. Five (13.8%) patients show mid-myocardial enhancement, followed by epicardial enhancement, which was seen in four (11.11%) patients. The values of post-gad T1 mapping values were significantly lower than pre-gad T1 mapping values. At a cut-off global native T1 mapping value of 1019 milliseconds had the sensitivity of 83.3% and specificity of 81.2% in detecting an abnormal T1 map. No significant association of MRI contrast enhancement with elevated ESR and CRP levels. There was no significant relation of myocardial T2 mapping values between NSAA and control groups. Conclusion Quantitative tissue characterization in the myocardium with native T1 mapping values help in the detection of cardiac involvement in patients with NSAA. T1 mapping may provide incremental value in the assessment of myocardial involvement in NSAA in addition to LGE imaging.

Computer-aided finite element model for biomechanical analysis of orthodontic aligners.

To design a finite element (FE) model that might facilitate understanding of the complex mechanical behavior of orthodontic aligners. The designed model was validated by comparing the generated forces - during 0.2-mm facio-lingual translation of upper left central incisor (Tooth 21) - with the values reported by experimental studies in literature. A 3D digital model, obtained from scanning of a typodont of upper jaw, was imported into 3-matic software for designing of aligners with different thicknesses: 0.4, 0.5, 0.6, 0.7mm. The model was exported to Marc/Mentat FE software. Suitable parameters for FE simulation were selected after a series of sensitivity analyses. Different element classes of the model and different rigidity values of the aligner were also investigated. The resultant maximum forces generated on facio-lingual translation of Tooth 21 were within the range of 1.3-18.3N. The force was direction-dependent, where lingual translation transmitted higher forces than facial translation. The force increases with increasing the thickness of the aligner, but not linearly. We found that the generated forces were almost directly proportional to the rigidity of the aligner. The contact normal stress map showed an uneven but almost repeatable distribution of stresses all over the facial surface and concentration of stresses at specific points. A validated FE model could reveal a lot about mechanical behavior of orthodontic aligners. Understanding the force systems of clear aligner by means of FE will facilitate better treatment planning and getting optimal outcomes.

A preliminary mapping of QTL qsg5.1 controlling seed germination in melon (Cucumis melo L.).

Melon (Cucumis melo L.) seed germination significantly affects its economic value. Cultivation of melon varieties with high germination ability and seedling vigor is beneficial in large-scale melon propagation. In this study, two melon genotypes differing in their germination ability, P5 with low and P10 with high germination ability, were used to identify the optimal seed germination conditions by evaluating different water immersion times and germination temperatures. The germination rate of the P5 and P10 parental genotypes and their segregating population, consisting of 358 F2:3 families, were evaluated for 2 years to identify their genetic basis. QTL analysis was performed on a high-density genetic map constructed using specific-locus amplified fragment sequencing (SLAF-seq). The germination rate of F1 and F2 populations treated with water immersion for 8 h at 28°C and measured at 48 h showed a normal distribution Genetic mapping carried out using the high-density genetic map revealed eight QTLs in chromosomes 2, 4, 5, 6, and 8 that control melon seed germination, of which 2020/2021-qsg5.1 was consistently significant in both years of experimentation. qsg5.1 explained 15.13% of the phenotypic variance with a LOD of 4.1. To fine map the candidate region of qsg5.1, eight cleaved amplified polymorphism sequence (CAPS) markers were used to construct a genetic map with another 421 F2 individual fruits. The major QTL qsg5.1 was located between SNP53 and SNP54 within a 55.96 Kb interval containing four genes. qRT-PCR gene expression analysis of the candidate genes showed that MELO3C031219.2 (Phosphorus transporter PHO-5) exhibited a significant difference in gene expression between the parental lines at 24, 32, and 48 h after germination, potentially being the underlying gene controlling melon seed germination. These results provide a theoretical basis for the molecular mechanisms controlling melon seed germination and can practically contribute to further improving germination to increase the propagation efficiency of commercial melon cultivars.