3D Medical Visualization Research Articles

Emergence of AI-Generated Multimedia: Visionary Physicists in Radiology Reincarnated.

AI-powered multimedia generation technologies, particularly video synthesis through stable diffusion and transformers, offer transformative potential for radiology education, communication, and visualization. This study explores various AI-generated multimedia categories, including image and video generation, as well as voice cloning, with a focus on video synthesis and future possibilities like scan-to-video generation. Utilizing tools such as Midjourney, RunwaymL Gen2, D-ID, and ElevenLabs, we aimed to reincarnate deceased influential physicists in radiology, demonstrating AI's capability to generate realistic content with accessible tools, fostering creativity and innovation in the radiology community. We created 440 images through 110 prompts using image-to-image generation, 22 videos via image-to-video generation, and two videos showcasing text-to-voice and voice cloning techniques from December 1-7, 2023. Realism decreased from image-to-image to image-to-video and voiceover-to-video generations, with the latter requiring adjustments for lip, mouth, and head movements without incorporating facial expressions, eye movement, or hand motions. Potential applications in radiology include improving and speeding up medical 3D visualization, as well as enhancing educational content, information delivery, patient interactions, and teleconsultations. The paper addresses limitations and ethical considerations associated with AI-generated content, emphasizing responsible use and interdisciplinary collaboration for further development. These technologies are rapidly evolving, and future versions are expected to address current challenges. The ongoing advancements in AI-generated multimedia have the potential to revolutionize various aspects of radiology practice, education, and patient care, opening new avenues for research and clinical applications in the field.

Chain code strategy for lossless storage and transfer of segmented binary medical data

Obtaining a 3D medical visualization is a tedious process requiring several processing steps (such as segmentation) and assigning various rendering parameters (such as color and opacity). Current systems use video/image exporting or snapshots to save results. Such vendor-dependent tools not only prevent the possibility of further interactions but also creates additional large-size data that is problematic to store in PACS over time and hard to transfer for teleradiology applications. To overcome, alternative strategies propose a representation of the visualizations, which only store segmentation masks that contains the binary form of segmented data. Unfortunately, existing compression methods are limited to effectively compress the volumetric data. In this study, lossless storage of binary segmented data is effectively performed by two newly-proposed chain code approaches. Particularly-two novel contributions are presented: 1. The dictionary of normalized angle difference is improved as a new chain symbol coding procedure, namely normalized angle difference, by adding new symbols to the dictionary aiming to generate a low-entropy symbol sequence for medical volumes. 2. A new volumetric approach that utilizes 26 symbols to encode volumetric data is developed. Each slice is visited, and the contour of the segmented object is codified such that eight different vectors for each slice (pointing to one of the four faces of each voxel, plus four towards one of its edges) are obtained. The developed methods are tested on diverse volumetric segmented data and compared to existing standards. It is shown that the proposed methods outperform well-established techniques.

Marching Cubes and Histogram Pyramids for 3D Medical Visualization.

This paper aims to implement histogram pyramids with marching cubes method for 3D medical volumetric rendering. The histogram pyramids are used for feature extraction by segmenting the image into the hierarchical order like the pyramid shape. The histogram pyramids can decrease the number of sparse matrixes that will occur during voxel manipulation. The important feature of the histogram pyramids is the direction of segments in the image. Then this feature will be used for connecting pixels (2D) to form up voxel (3D) during marching cubes implementation. The proposed method is fast and easy to implement and it also produces a smooth result (compared to the traditional marching cubes technique). The experimental results show the time consuming for generating 3D model can be reduced by 15.59% in average. The paper also shows the comparison between the surface rendering using the traditional marching cubes and the marching cubes with histogram pyramids. Therefore, for the volumetric rendering such as 3D medical models and terrains where a large number of lookups in 3D grids are performed, this method is a particularly good choice for generating the smooth surface of 3D object.

Pores, Pimples and Pathologies: 3D Capture and Detailing of the Human Skin for 3D Medical Visualisation and Fabrication.

Three-dimensional (3D) scanning of the human skin for 3D medical visualisation and printing does not often produce the desired results due to a number of factors including the specularity of human skin, difficulties in scanning fine structures such as the hair and the capabilities of the scanning technologies utilised. Some additional 3D modelling may be required to make the surfaces more suitable for use in the production of anatomical and medical teaching resources, computerised facial depiction and design of bespoke prostheses. Three-dimensional scanned surfaces can be enhanced through digital sculpting and embossing of high-resolution photographs of the human skin.

Labels on Levels: Labeling of Multi-Scale Multi-Instance and Crowded 3D Biological Environments.

Labeling is intrinsically important for exploring and understanding complex environments and models in a variety of domains. We present a method for interactive labeling of crowded 3D scenes containing very many instances of objects spanning multiple scales in size. In contrast to previous labeling methods, we target cases where many instances of dozens of types are present and where the hierarchical structure of the objects in the scene presents an opportunity to choose the most suitable level for each placed label. Our solution builds on and goes beyond labeling techniques in medical 3D visualization, cartography, and biological illustrations from books and prints. In contrast to these techniques, the main characteristics of our new technique are: 1) a novel way of labeling objects as part of a bigger structure when appropriate, 2) visual clutter reduction by labeling only representative instances for each type of an object, and a strategy of selecting those. The appropriate level of label is chosen by analyzing the scene’s depth buffer and the scene objects’ hierarchy tree. We address the topic of communicating the parent-children relationship between labels by employing visual hierarchy concepts adapted from graphic design. Selecting representative instances considers several criteria tailored to the character of the data and is combined with a greedy optimization approach. We demonstrate the usage of our method with models from mesoscale biology where these two characteristics—multi-scale and multi-instance—are abundant, along with the fact that these scenes are extraordinarily dense.

Development and use of augmented reality and 3D printing in consulting patient with complex skull base cholesteatoma

ABSTRACTTemporal bone and skull base anatomy is complex and can pose difficulties in patient’s perception of disease and anatomy, perceived risks, and complications of surgery. We wish to demonstrate the development and use of augmented reality (AR) technology and three-dimensional (3D) printing to assist in preoperative patient consultation in the case of a complex skull base cholesteatoma. A series of 3D models were constructed from a patient’s petrous temporal bone computed tomography (CT) scans using CAD (computer-aided design) software to display the cholesteatoma affected temporal bone and related structures. Supplementary labels and titles were also created. A two-dimensional (2D) image was created as an AR recognition marker. Both 3D and 2D assets were uploaded, linked in an AR development platform called Hyperspaces which was then published to the Hyperspaces cloud server to build an AR application for free access using a predefined keyword on an iOS mobile device. Patient’s temporal bone was also fabricated through a fused deposition modelling 3D printer using polylactic acid filament for patient consultation. AR and 3D printing allow patient-specific clinical information and complexities to be made readily available to the patient and doctor at low cost, and aid in understanding complex skull base anatomy and progressive disease such as cholesteatoma. The advancement of AR and 3D printing technologies are making complex and patient-specific 3D medical data visualisation feasible and tangible on a mobile device and in hand. Thus, these technologies can be used as an invaluable patient education and counselling tool offering a powerful medium in specialties where difficult anatomical challenges are encountered.Abbreviations: ABS: acrylonitrile–butadiene–styrene; AM: additive manufacturing; AR: augmented reality; CAD: computer-aided design; CT: computed tomography; DICOM: Digital Imaging and Communication in Medicine; FBX: Filmbox; FDM: fused deposition modelling; JPEG: joint photographic experts group; MRI: magnetic resonance imaging; PLA: polylactic acid; ROI: region of interest; STL: Stereolithography

Systematic Parameterization, Storage, and Representation of Volumetric DICOM Data.

Tomographic medical imaging systems produce hundreds to thousands of slices, enabling three-dimensional (3D) analysis. Radiologists process these images through various tools and techniques in order to generate 3D renderings for various applications, such as surgical planning, medical education, and volumetric measurements. To save and store these visualizations, current systems use snapshots or video exporting, which prevents further optimizations and requires the storage of significant additional data. The Grayscale Softcopy Presentation State extension of the Digital Imaging and Communications in Medicine (DICOM) standard resolves this issue for two-dimensional (2D) data by introducing an extensive set of parameters, namely 2D Presentation States (2DPR), that describe how an image should be displayed. 2DPR allows storing these parameters instead of storing parameter applied images, which cause unnecessary duplication of the image data. Since there is currently no corresponding extension for 3D data, in this study, a DICOM-compliant object called 3D presentation states (3DPR) is proposed for the parameterization and storage of 3D medical volumes. To accomplish this, the 3D medical visualization process is divided into four tasks, namely pre-processing, segmentation, post-processing, and rendering. The important parameters of each task are determined. Special focus is given to the compression of segmented data, parameterization of the rendering process, and DICOM-compliant implementation of the 3DPR object. The use of 3DPR was tested in a radiology department on three clinical cases, which require multiple segmentations and visualizations during the workflow of radiologists. The results show that 3DPR can effectively simplify the workload of physicians by directly regenerating 3D renderings without repeating intermediate tasks, increase efficiency by preserving all user interactions, and provide efficient storage as well as transfer of visualized data.Electronic supplementary materialThe online version of this article (doi:10.1007/s40846-015-0097-5) contains supplementary material, which is available to authorized users.

Hand shape classification using depth data for unconstrained 3D interaction

In this paper, we introduce a novel method for view-independent hand pose recognition from depth data. The proposed approach, which does not rely on color information, provides an estimation of the shape and orientation of the user's hand without constraining him/her to maintain a fixed position in the 3D space. We use principal component analysis to estimate the hand orientation in space, Flusser moment invariants as image features and two SVM-RBF classifiers for visual recognition. Moreover, we describe a novel weighting method that takes advantage of the orientation and velocity of the user's hand to assign a score to each hand shape hypothesis. The complete processing chain is described and evaluated in terms of real-time performance and classification accuracy. As a case study, it has also been integrated into a touchless interface for 3D medical visualization, which allows users to manipulate 3D anatomical parts with up to six degrees of freedom. Furthermore, the paper discusses the results of a user study aimed at assessing if using hand velocity as an indicator of the user's intentionality in changing hand posture results in an overall gain in the classification accuracy. The experimental results show that, especially in the presence of out-of-plane rotations of the hand, the introduction of the velocity-based weighting method produces a significant increase in the pose recognition accuracy.

Segmentation and reconstruction of hepatic veins and intrahepatic portal vein based on the coronal sectional anatomic dataset

Three-dimensional (3D) reconstruction of intrahepatic vessels is very useful in visualizing the complex anatomy of hepatic veins and intrahepatic portal vein. It also provides a 3D anatomic basis for diagnostic imaging and surgical operation on the liver. In the present study, we built a 3D digitized model of hepatic veins and intrahepatic portal vein based on the coronal sectional anatomic dataset of the liver. The dataset was obtained using the digital freezing milling technique. The pre-reconstructed structures were identified and extracted, and then were segmented by the method of manual intervention. The digitized model of hepatic veins and intrahepatic portal vein was established using 3D medical visualization software. This model facilitated a continuous and dynamic displaying of the hepatic veins and intrahepatic portal vein at different orientations, which demonstrated the complicated relationship of adjacent hepatic veins and intrahepatic portal vein realistically in the 3D space. This study indicated that high-quality 2D images, precise data segmentation, and suitable 3D reconstruction methods ensured the reality and accuracy of the digital visualized model of hepatic veins and intrahepatic portal vein.

An interactive 3D medical visualization system based on a light field display

We present a prototype medical data visualization system exploiting a light field display and custom direct volume rendering techniques to enhance understanding of massive volumetric data, such as CT, MRI, and PET scans. The system can be integrated with standard medical image archives and extends the capabilities of current radiology workstations by supporting real-time rendering of volumes of potentially unlimited size on light field displays generating dynamic observer-independent light fields. The system allows multiple untracked naked-eye users in a sufficiently large interaction area to coherently perceive rendered volumes as real objects, with stereo and motion parallax cues. In this way, an effective collaborative analysis of volumetric data can be achieved. Evaluation tests demonstrate the usefulness of the generated depth cues and the improved performance in understanding complex spatial structures with respect to standard techniques.