3D Mesh Geometry Research Articles

Development and Validation of a Novel Methodological Pipeline to Integrate Neuroimaging and Photogrammetry for Immersive 3D Cadaveric Neurosurgical Simulation.

BackgroundVisualizing and comprehending 3-dimensional (3D) neuroanatomy is challenging. Cadaver dissection is limited by low availability, high cost, and the need for specialized facilities. New technologies, including 3D rendering of neuroimaging, 3D pictures, and 3D videos, are filling this gap and facilitating learning, but they also have limitations. This proof-of-concept study explored the feasibility of combining the spatial accuracy of 3D reconstructed neuroimaging data with realistic texture and fine anatomical details from 3D photogrammetry to create high-fidelity cadaveric neurosurgical simulations.MethodsFour fixed and injected cadaver heads underwent neuroimaging. To create 3D virtual models, surfaces were rendered using magnetic resonance imaging (MRI) and computed tomography (CT) scans, and segmented anatomical structures were created. A stepwise pterional craniotomy procedure was performed with synchronous neuronavigation and photogrammetry data collection. All points acquired in 3D navigational space were imported and registered in a 3D virtual model space. A novel machine learning-assisted monocular-depth estimation tool was used to create 3D reconstructions of 2-dimensional (2D) photographs. Depth maps were converted into 3D mesh geometry, which was merged with the 3D virtual model’s brain surface anatomy to test its accuracy. Quantitative measurements were used to validate the spatial accuracy of 3D reconstructions of different techniques.ResultsSuccessful multilayered 3D virtual models were created using volumetric neuroimaging data. The monocular-depth estimation technique created qualitatively accurate 3D representations of photographs. When 2 models were merged, 63% of surface maps were perfectly matched (mean [SD] deviation 0.7 ± 1.9 mm; range −7 to 7 mm). Maximal distortions were observed at the epicenter and toward the edges of the imaged surfaces. Virtual 3D models provided accurate virtual measurements (margin of error <1.5 mm) as validated by cross-measurements performed in a real-world setting.ConclusionThe novel technique of co-registering neuroimaging and photogrammetry-based 3D models can (1) substantially supplement anatomical knowledge by adding detail and texture to 3D virtual models, (2) meaningfully improve the spatial accuracy of 3D photogrammetry, (3) allow for accurate quantitative measurements without the need for actual dissection, (4) digitalize the complete surface anatomy of a cadaver, and (5) be used in realistic surgical simulations to improve neurosurgical education.

Foids

We present a bio-inspired fish simulation platform, which we call "Foids", to generate realistic synthetic datasets for an use in computer vision algorithm training. This is a first-of-its-kind synthetic dataset platform for fish, which generates all the 3D scenes just with a simulation. One of the major challenges in deep learning based computer vision is the preparation of the annotated dataset. It is already hard to collect a good quality video dataset with enough variations; moreover, it is a painful process to annotate a sufficiently large video dataset frame by frame. This is especially true when it comes to a fish dataset because it is difficult to set up a camera underwater and the number of fish (target objects) in the scene can range up to 30,000 in a fish cage on a fish farm. All of these fish need to be annotated with labels such as a bounding box or silhouette, which can take hours to complete manually, even for only a few minutes of video. We solve this challenge by introducing a realistic synthetic dataset generation platform that incorporates details of biology and ecology studied in the aquaculture field. Because it is a simulated scene, it is easy to generate the scene data with annotation labels from the 3D mesh geometry data and transformation matrix. To this end, we develop an automated fish counting system utilizing the part of synthetic dataset that shows comparable counting accuracy to human eyes, which reduces the time compared to the manual process, and reduces physical injuries sustained by the fish.

Validation of Close-Range Photogrammetry for Architectural and Archaeological Heritage: Analysis of Point Density and 3D Mesh Geometry

The 3D digitization and Building Information Modeling (BIM), which is based on parametric objects, have considerably advanced by developing massive data capture techniques. Thus, reverse engineering currently plays a major role as these technologies capture accurately and efficiently the geometry, color and textures of complex architectural, archaeological and cultural heritage. This paper aims to validate close-range Structure from Motion (SfM) for heritage by analyzing the point density and the 3D mesh geometry in comparison with Terrestrial Laser Scanning (TLS). The accuracy of the results and the geometry mainly depends on the processing performed on the point set. Therefore, these two variables are significant in the 3D reconstruction of heritage buildings. This paper focuses on a 15th century case study in Seville (Spain): the main façade of Casa de Pilatos. Ten SfM surveys were carried out varying the capture method (simple and stereoscopic) and the number of shots, distances, orientation and procedure. A mathematical analysis is proposed to verify the point spatial resolution and the accuracy of the 3D model geometry by section profiles in SfM data. SfM achieved acceptable accuracy levels to generate 3D meshes despite disordered shots and the number of images. Hence, stereoscopic photography using new instruments improved the results of close-range photogrammetry while reducing the required number of photographs.

Depth texture synthesis for high-resolution reconstruction of large scenes

Large scenes such as building facades and other architectural constructions often contain repeating elements such as identical windows and brick patterns. In this paper, we present a novel approach that improves the resolution and geometry of 3D meshes of large scenes with such repeating elements. By leveraging structure from motion reconstruction and an off-the-shelf depth sensor, our approach captures a small sample of the scene in high resolution and automatically extends that information to similar regions of the scene. Using RGB and SfM depth information as a guide and simple geometric primitives as canvas, our approach extends the high-resolution mesh by exploiting powerful, image-based texture synthesis approaches. The final results improve on standard SfM reconstruction with higher detail. Our approach benefits from reduced manual labor as opposed to full RGBD reconstruction, and can be done much more cheaply than with LiDAR-based solutions.

Masked 3D conditional generative adversarial network for rock mesh generation

3D-GAN (Wu et al. in: Advances in Neural Information Processing Systems, pp. 82–90, 2016) has been introduced as a novel way to generate 3D models. In this paper, we propose a 3D-Masked-CGAN approach to apply in the generation of irregular 3D mesh geometry such as rocks. While there are many ways to generate 3D objects, the generation of irregular 3D models has its own peculiarity. To make a model realistic is extremely time-consuming and in high cost. In order to control the shape of generated 3D models, we extend 3D-GAN by adding conditional information into both the generator and discriminator. It is shown that that this model can generate 3D rock models with effective control over the shapes of generated models.

A Novel Approach For Ankle Foot Orthosis Developed By Three Dimensional Technologies

This study presents a novel approach for testing mechanical properties of medical orthosis developed by three dimensional (3D) technologies. A hand-held type 3D laser scanner is used for generating 3D mesh geometry directly from patient’s limb. Subsequently 3D printable orthotic design is produced from crude input model by means of Computer Aided Design (CAD) software. Fused Deposition Modelling (FDM) method in Additive Manufacturing (AM) technologies is used to fabricate the 3D printable Ankle Foot Orthosis (AFO) prototype in order to test the mechanical properties on printout. According to test results, printed Acrylonitrile Butadiene Styrene (ABS) AFO prototype has sufficient elasticity modulus and durability for patient-specific medical device manufactured by the 3D technologies.

Refining Geometry from Depth Sensors using IR Shading Images

We propose a method to refine geometry of 3D meshes from a consumer level depth camera, e.g. Kinect, by exploiting shading cues captured from an infrared (IR) camera. A major benefit to using an IR camera instead of an RGB camera is that the IR images captured are narrow band images that filter out most undesired ambient light, which makes our system robust against natural indoor illumination. Moreover, for many natural objects with colorful textures in the visible spectrum, the subjects appear to have a uniform albedo in the IR spectrum. Based on our analyses on the IR projector light of the Kinect, we define a near light source IR shading model that describes the captured intensity as a function of surface normals, albedo, lighting direction, and distance between light source and surface points. To resolve the ambiguity in our model between the normals and distances, we utilize an initial 3D mesh from the Kinect fusion and multi-view information to reliably estimate surface details that were not captured and reconstructed by the Kinect fusion. Our approach directly operates on the mesh model for geometry refinement. We ran experiments on our algorithm for geometries captured by both the Kinect I and Kinect II, as the depth acquisition in Kinect I is based on a structured-light technique and that of the Kinect II is based on a time-of-flight technology. The effectiveness of our approach is demonstrated through several challenging real-world examples. We have also performed a user study to evaluate the quality of the mesh models before and after our refinements.

Realistic rendering of organ for surgery simulator

Realistic rendering of the organ surface appearance is necessary for high quality surgery simulator. It can greatly enhance visual realism and hence the overall quality of the VR simulators can be improved. This paper studied the realistic rendering methods of the organ surface, including organ texture synthesis, texture mapping and lighting computation for modeling mucous-covering of organ surfaces. Two different organ texture synthesis methods were presented: procedural texture generation based on Perlin noise, and applying an example synthesis based on an Ashikhmin algorithm. All kinds of organ textures were synthesized effectively. With regard to the visibility of seams and reducing the distortion, the created textures can be mapped to 3D mesh geometry by mesh parameterization. Blending between the different textures of patches reduced discontinuities. A kind of lighting computation method was described for modeling translucent mucous covering, in which incoming light is decomposed into its two components of surface reflection and sub-surface scattering that are then deal with respectively. Experimental results show that the proposed method is both feasible and effective.

Compression of 3D Mesh Geometry and Vertex Attributes for Mobile Graphics

This paper presents a compression scheme for mesh geometry, which is suitable for mobile graphics. The main focus is to enable real-time decoding of compressed vertex positions while providing reasonable compression ratios. Our scheme is based on local quantization of vertex positions with mesh partitioning. To prevent visual seams along the partitioning boundaries, we constrain the locally quantized cells of all mesh partitions to have the same size and aligned local axes. We propose a mesh partitioning algorithm to minimize the size of locally quantized cells, which relates to the distortion of a restored mesh. Vertex coordinates are stored in main memory and transmitted to graphics hardware for rendering in the quantized form, saving memory space and system bus bandwidth. Decoding operation is combined with model geometry transformation, and the only overhead to restore vertex positions is one matrix multiplication for each mesh partition. In our experiments, a 32-bit floating point vertex coordinate is quantized into an 8-bit integer, which is the smallest data size supported in a mobile graphics library. With this setting, the distortions of the restored meshes are comparable to 11-bit global quantization of vertex coordinates. We also apply the proposed approach to compression of vertex attributes, such as vertex normals and texture coordinates, and show that gains similar to vertex geometry can be obtained through local quantization with mesh partitioning.

패치 영상의 효율적 압축을 위한 전처리 방법

텍스처 영상 압축은 JPEG과 같은 영상 압축 표준을 사용한다. 일반 텍스처 영상에는 JPEG으로도 좋은 압축 성능을 내지만 패치 텍스처 영상은 이러한 영상 압축 기술로 압축했을 때 좋은 압축 성능을 보이지 못하였다. 이를 보완하고 압축 성능을 높이기 위해 기존의 패치 텍스처 영상에 주변 화소간에 색차를 줄이기 위한 전처리 기술을 소개하고, 제안된 기술과 JPEG 압축을 순서대로 적용하였을 때 압축 성능이 23%에서 45%까지 향상됨을 보이고자 한다. In mapping a texture image into a 3D mesh model for photo-realistic graphic applications, the compression of texture image is as important as geometry of 3D mesh. Typically, the size of the (compressed) texture image of 3D model is comparable to that of the (compressed) 3D mesh geometry. Most 3D model compression techniques are to compress the 3D mesh geometry, rather than to compress the texture image. Well-known image compression standards (i.e., JPEG) have been extensively used for texture image compression. However, such techniques are not so efficient when it comes to compress an image with texture patches, since the patches are little correlated. In this paper, we proposed a preprocessing method to substantially improve the compression efficiency of texture compression. From the experimental results, the proposed method was shown to be efficient in compression with a bit-saving from 23% to 45%.

3D mesh geometry filtering algorithms for progressive transmission schemes

A number of static and multi-resolution methods have been introduced in recent years to compress 3D meshes. In most of these methods, the connectivity information is encoded without loss of information, but user-controllable loss of information is tolerated while compressing the geometry and property data. All these methods are very efficient at compressing the connectivity information, in some cases to a fraction of a bit per vertex, but the geometry and property data typically occupies much more room in the compressed bitstream than the compressed connectivity data. In this paper, we investigate the use of polynomial linear filtering as studied in the Refs. [Taubin G. A signal processing approach to fair surface design. Computer Graphics Proc., Annual Conference Series 1995;351–358; Taubin G, Zhang T, Golub G. Optimal surface smoothing as filter design. IBM Research report RC–20,404, 1996], as a global predictor for the geometry data of a 3D mesh in multi-resolution 3D geometry compression schemes. Rather than introducing a new method to encode the multi-resolution connectivity information, we choose one of the efficient existing schemes depending on the structure of the multi-resolution data. After encoding the geometry of the lowest level of detail with an existing scheme, the geometry of each subsequent level of detail is predicted by applying a polynomial filter to the geometry of its predecessor lifted to the connectivity of the current level. The polynomial filter is designed to minimize the l2-norm of the approximation error but other norms can be used as well. Three properties of the filtered mesh are studied next: accuracy, robustness and compression ratio. The Zeroth Order Filter (unit polynomial) is found to have the best compression ratio. But higher order filters achieve better accuracy and robustness properties at the price of a slight decrease of the compression ratio.