2D Projection Research Articles

Robotized technologies for enhanced shipyard operations: challenges and solutions

Large component manufacturing relies heavily on manual operations and human workers. Human-centric solutions can preserve industry-specific knowledge, extend capabilities, and improve job performance. Three robotized technologies were developed for shipyard operations: ABB™ and KUKA™ robot hand-guiding systems (HGS), a lightweight collaborative system for plasma cutting, and a cost-effective 3D projection system for retrofitting. These technologies were developed at the open didactic factory, which served as platforms for rapid technological advancement. The HGS was integrated with ABB™ and KUKA™, and the 3D projection technology and lightweight collaborative system offered a cost-effective solution for small and medium shipyards. However, transitioning to non-flat surfaces presents challenges due to geometric variations and discrepancies between the computer-aided design model and the actual component.

Beyond the Label Itself: Latent Labels Enhance Semi-supervised Point Cloud Panoptic Segmentation

As the exorbitant expense of labeling autopilot datasets and the growing trend of utilizing unlabeled data, semi-supervised segmentation on point clouds becomes increasingly imperative. Intuitively, finding out more ``unspoken words'' (i.e., latent instance information) beyond the label itself should be helpful to improve performance. In this paper, we discover two types of latent labels behind the displayed label embedded in LiDAR and image data. First, in the LiDAR Branch, we propose a novel augmentation, Cylinder-Mix, which is able to augment more yet reliable samples for training. Second, in the Image Branch, we propose the Instance Position-scale Learning (IPSL) Module to learn and fuse the information of instance position and scale, which is from a 2D pre-trained detector and a type of latent label obtained from 3D to 2D projection. Finally, the two latent labels are embedded into the multi-modal panoptic segmentation network. The ablation of the IPSL module demonstrates its robust adaptability, and the experiments evaluated on SemanticKITTI and nuScenes demonstrate that our model outperforms the state-of-the-art method, LaserMix.

YOLO2U-Net: Detection-guided 3D instance segmentation for microscopy

Microscopy imaging techniques are instrumental for characterization and analysis of biological structures. As these techniques typically render 3D visualization of cells by stacking 2D projections, issues such as out-of-plane excitation and low resolution in the z-axis may pose challenges (even for human experts) to detect individual cells in 3D volumes as these non-overlapping cells may appear as overlapping. A comprehensive method for accurate 3D instance segmentation of cells in the brain tissue is introduced here. The proposed method combines the 2D YOLO detection method with a multi-view fusion algorithm to construct a 3D localization of the cells. Next, the 3D bounding boxes along with the data volume are input to a 3D U-Net network that is designed to segment the primary cell in each 3D bounding box, and in turn, to carry out instance segmentation of cells in the entire volume. The promising performance of the proposed method is shown in comparison with current deep learning-based 3D instance segmentation methods.

Direct May Not Be the Best: An Incremental Evolution View of Pose Generation

Pose diversity is an inherent representative characteristic of 2D images. Due to the 3D to 2D projection mechanism, there is evident content discrepancy among distinct pose images. This is the main obstacle bothering pose transformation related researches. To deal with this challenge, we propose a fine-grained incremental evolution centered pose generation framework, rather than traditional direct one-to-one in a rush. Since proposed approach actually bypasses the theoretical difficulty of directly modeling dramatic non-linear variation, the incurred content distortion and blurring could be effectively constrained, at the same time the various individual pose details, especially clothes texture, could be precisely maintained. In order to systematically guide the evolution course, both global and incremental evolution constraints are elaborately designed and merged into the overall framework. And a novel triple-path knowledge fusion structure is worked out to take full advantage of all available valuable knowledge to conduct high-quality pose synthesis. In addition, our framework could generate a series of valuable by-products, namely the various intermediate poses. Extensive experiments have been conducted to verify the effectiveness of the proposed approach. Code is available at https://github.com/Xiaofei-CN/Incremental-Evolution-Pose-Generation.

Binding-Free Taste Visualization with Plasmonic Metasurfaces.

Taste sensors using photonics, termed artificial photonic tongues, have emerged as a promising platform for intuitive taste discrimination. However, the need for complex binding protocols for each taste profile limits their applicability to a narrow range of taste molecules. Here, we introduce an intriguing "binding-free" approach to molecular taste sensing using plasmonics, eliminating the requirement for physical or chemical binding protocols. We develop a wafer-scale plasmonic metasurface constructed by coating metallic nanoparticles in a scalable manner onto a metallic mirror. This metasurface functions to detect molecular refractive indices and surface tensions via 2D projection optical images of an array of liquid droplets containing the taste molecules on top, which can immediately visualize and distinguish between the five basic tastes of molecules (including their mixtures) as well as other additional spicy and alcoholic tastes. We anticipate that this intuitive and rapid taste-sensing approach has the potential to establish a user-friendly and portable taste-sensing platform.

Poincaré sphere trajectory encoding metasurfaces based on generalized Malus’ law

As a fundamental property of light, polarization serves as an excellent information encoding carrier, playing significant roles in many optical applications, including liquid crystal displays, polarization imaging, optical computation and encryption. However, conventional polarization information encoding schemes based on Malus’ law usually consider 1D polarization projections on a linear basis, implying that their encoding flexibility is largely limited. Here, we propose a Poincaré sphere (PS) trajectory encoding approach with metasurfaces that leverages a generalized form of Malus’ law governing universal 2D projections between arbitrary elliptical polarization pairs spanning the entire PS. Arbitrary polarization encodings are realized by engineering PS trajectories governed by either arbitrary analytic functions or aligned modulation grids of interest, leading to versatile polarization image transformation functionalities, including histogram stretching, thresholding and image encryption within non-orthogonal PS loci. Our work significantly expands the encoding dimensionality of polarization information, unveiling new opportunities for metasurfaces in polarization optics for both quantum and classical regimes.

Three-dimensional image authentication from multi-view images.

Three-dimensional (3D) optical authentication is important for modern information security. Existing 3D optical authentication methods rely on integral imaging devices, necessitating meticulous calibration and incurring high transmission overhead. To streamline the acquisition of 3D information, this paper introduces a novel 3D optical authentication approach, to the best of our knowledge, based on the construction of 3D data from multi-view images. The proposed method simplifies 3D projection by generating fixed-viewpoint elemental images, eliminating the need for additional viewpoint information during transmission and authentication. Compressed sensing is used for compression during transmission, and a deep learning network is designed for 3D reconstruction, enhancing the recovery. Experimental outcomes confirm the efficiency of our proposed approach for 3D authentication across diverse datasets.

Deformable registration of preoperative MR and intraoperative long-length tomosynthesis images for guidance of spine surgery via image synthesis

PurposeImproved integration and use of preoperative imaging during surgery hold significant potential for enhancing treatment planning and instrument guidance through surgical navigation. Despite its prevalent use in diagnostic settings, MR imaging is rarely used for navigation in spine surgery. This study aims to leverage MR imaging for intraoperative visualization of spine anatomy, particularly in cases where CT imaging is unavailable or when minimizing radiation exposure is essential, such as in pediatric surgery. MethodsThis work presents a method for deformable 3D-2D registration of preoperative MR images with a novel intraoperative long-length tomosynthesis imaging modality (viz., Long-Film [LF]). A conditional generative adversarial network is used to translate MR images to an intermediate bone image suitable for registration, followed by a model-based 3D-2D registration algorithm to deformably map the synthesized images to LF images. The algorithm’s performance was evaluated on cadaveric specimens with implanted markers and controlled deformation, and in clinical images of patients undergoing spine surgery as part of a large-scale clinical study on LF imaging. ResultsThe proposed method yielded a median 2D projection distance error of 2.0 mm (interquartile range [IQR]: 1.1–3.3 mm) and a 3D target registration error of 1.5 mm (IQR: 0.8–2.1 mm) in cadaver studies. Notably, the multi-scale approach exhibited significantly higher accuracy compared to rigid solutions and effectively managed the challenges posed by piecewise rigid spine deformation. The robustness and consistency of the method were evaluated on clinical images, yielding no outliers on vertebrae without surgical instrumentation and 3% outliers on vertebrae with instrumentation. ConclusionsThis work constitutes the first reported approach for deformable MR to LF registration based on deep image synthesis. The proposed framework provides access to the preoperative annotations and planning information during surgery and enables surgical navigation within the context of MR images and/or dual-plane LF images.

Estimation of radiographic joint space of the trapeziometacarpal joint with computed tomographic validation.

Joint space width (JSW) is a common metric used to evaluate joint structure on plain radiographs. For the hand, quantitative techniques are available for evaluation of the JSW of finger joints; however, such techniques have been difficult to establish for the trapeziometacarpal (TMC) joint. This study aimed to develop a validated method for measuring the radiographic joint space of the healthy TMC joint. Computed tomographic scans were taken of 15 cadaveric hands. The location of a JSW analysis region on the articular surface of the first metacarpal was established in 3D space and standardized in a 2D projection. The standardized region was applied to simulated radiographic images. A correction factor was defined as the ratio of the CT-based and radiograph-based joint space measurements. Leave-one-out validation was used to correct the radiograph-based measurements. A t-test was used to evaluate the difference between CT-based and corrected radiograph-based measurements (). The CT-based and radiograph-based measurements of JSW were and , respectively. The correction factor for radiograph-based joint space was . Before correction, the difference between the CT-based and radiograph-based joint space was 1.43mm [95% CI: ; ]. After correction, the difference was [95% CI: ; ]. Corrected measurements of radiographic TMC JSW agreed well with CT-measured JSW. With in-vivo validation, the developed methodology has potential for automated and accurate radiographic measurement of TMC JSW.

Global and local multi-modal feature mutual learning for retinal vessel segmentation

Research on optical coherence tomography angiography (OCTA) images has received extensive attention in recent years since it provides more detailed information about retinal structures. The automatic segmentation of retinal vessel (RV) has become one of the key issues in the quantification of retinal indicators. To this end, there are various methods proposed with cutting-edge designs and techniques in the literature. However, most of them only learn features from single-modal data, despite the potential relation between data from different modalities. Clinically, 2D projection maps are more convenient for doctors to observe. Nevertheless, 3D volumes preserve the intrinsic retinal structure. We thus propose a novel multi-modal feature mutual learning framework that contains local mutual learning and global mutual learning capturing 2D and 3D information. In the framework, the 3D model and 2D model learn collaboratively and teach each other throughout the training process. Experimental results show that our method outperforms previous deep-learning methods in RV segmentation. The generalization experiments on the ROSE dataset demonstrate the portability and scalability of the proposed framework.

Analysis of Students' Design Thinking in Creating 3D Creative Works Utilizing Mobile Technology within the Framework of Local Wisdom

This study investigated students' design thinking in creating 3D creative works using mobile technology in the framework of local wisdom. By employing a mixed-methods approach, the study compares the design thinking abilities of students who engaged with mobile technology in the context of local wisdom with those who donot. Quantitative analysis, involving sixty students, revealed that the experimental group, exposed to mobile technology and local wisdom, demonstrated significantly higher competencies in design thinking, particularly in areas such as comfort with uncertainty, human-centeredness, and collaborative work. Qualitative insights further illuminated the enriched learning experience, showcasing students' deepened engagement and connection to cultural heritage through the integration of local wisdom into their 3D creative projects. The findings underscore the pivotal role of mobile technology and local wisdom in fostering innovative design thinking, suggesting that this approach not only facilitates creative problem-solving but also instills a sense of cultural identity and pride among students. Despite its focused educational context and limited sample size, the study suggested broader implications for curriculum development, recommending the incorporation of these elements into design and creative courses to enhance students’ design skills and cultural understanding.

LA-Net: layer attention network for 3D-to-2D retinal vessel segmentation in OCTA images

Objective. Retinal vessel segmentation from optical coherence tomography angiography (OCTA) volumes is significant in analyzing blood supply structures and the diagnosing ophthalmic diseases. However, accurate retinal vessel segmentation in 3D OCTA remains challenging due to the interference of choroidal blood flow signals and the variations in retinal vessel structure. Approach. This paper proposes a layer attention network (LA-Net) for 3D-to-2D retinal vessel segmentation. The network comprises a 3D projection path and a 2D segmentation path. The key component in the 3D path is the proposed multi-scale layer attention module, which effectively learns the layer features of OCT and OCTA to attend to the retinal vessel layer while suppressing the choroidal vessel layer. This module also efficiently captures 3D multi-scale information for improved semantic understanding during projection. In the 2D path, a reverse boundary attention module is introduced to explore and preserve boundary and shape features of retinal vessels by focusing on non-salient regions in deep features. Main results. Experimental results in two subsets of the OCTA-500 dataset showed that our method achieves advanced segmentation performance with Dice similarity coefficients of 93.04% and 89.74%, respectively. Significance. The proposed network provides reliable 3D-to-2D segmentation of retinal vessels, with potential for application in various segmentation tasks that involve projecting the input image. Implementation code: https://github.com/y8421036/LA-Net.

Topology observing 3D device reconstruction from continuous-sweep limited angle fluoroscopy.

Minimally invasive procedures usually require navigating a microcatheter and guidewire through endoluminal structures such as blood vessels and airways to sites of the disease. For numerous clinical applications, two-dimensional (2D) fluoroscopy is the primary modality used for real-time image guidance during navigation. However, 2D imaging can pose challenges for navigation in complex structures. Real-time 3D visualization of devices within the anatomic context could provide considerable benefits for these procedures. Continuous-sweep limited angle (CLA) fluoroscopy has recently been proposed to provide a compromise between conventional rotational 3D acquisitions and real-timefluoroscopy. The purpose of this work was to develop and evaluate a noniterative 3D device reconstruction approach for CLA fluoroscopy acquisitions, which takes into account endoluminal topology to avoid impossible paths between disconnectedbranches. The algorithm relies on a static 3D roadmap (RM) of vessels or airways, which may be generated from conventional cone beam CT (CBCT) acquisitions prior to navigation. The RM is converted to a graph representation describing its topology. During catheter navigation, the device is segmented from the live 2D projection images using a deep learning approach from which the centerlines are extracted. Rays from the focal spot to detector pixels representing 2D device points are identified and intersections with the RM are computed. Based on the RM graph, a subset of line segments is selected as candidates to exclude device paths through disconnected branches of the RM. Depth localization for each point along the device is then performed by finding the point closest to the previous 3D reconstruction along the candidate segments. This process is repeated as the projection angle changes for each CLA image frame. The approach was evaluated in a phantom study in which a catheter and guidewire were navigated along five pathways within a complex vessel phantom. The result was compared to static cCBCT acquisitions of the device in the finalposition. The average root mean squared 3D distance between CLA reconstruction and reference centerline was mm. The Euclidean distance at the device tip was mm. The correct pathway was identified during reconstruction in of frames ( ). The percentage of 3D device points reconstructed inside the 3D roadmap was with an average distance of mm between the device points outside the roadmap and the nearest point within theroadmap. This study demonstrates the feasibility of reconstructing curvilinear devices such as catheters and guidewires during endoluminal procedures including intravascular and transbronchial interventions using a noniterative reconstruction approach for CLA fluoroscopy. This approach could improve device navigation in cases where the structure of vessels or airways is complex and includes overlappingbranches.

Broad Elevation Projection Super-Resolution Ultrasound (BEP-SRUS) Imaging With a 1-D Unfocused Linear Array.

Super-resolution ultrasound (SRUS) through localizing spatially isolated microbubbles (MBs) has been demonstrated to overcome the wave diffraction limit and reveal the microvascular structure and flow information at the microscopic scale. However, 3-D SRUS imaging remains a challenge due to the fabrication and computational complexity of 2-D matrix array probes. Inspired by X-ray radiography which can present information within a volume in a single projection image with much simpler hardware than X-ray computerized tomography (CT), this study investigates the feasibility of broad elevation projection super-resolution (BEP-SR) ultrasound using a 1-D unfocused linear array. Both simulation and in vitro experiments were conducted on 3-D microvessel phantoms. In vivo demonstration was done on the Rabbit kidney. Data from a 1-D linear array with and without an elevational focus were synthesized by summing up row signals acquired from a 2-D matrix array with and without delays. A full 3-D reconstruction was also generated as the reference, using the same data of the 2-D matrix array but without summing row signals. Results show that using an unfocused 1-D array probe, BEP-SR can capture significantly more information within a volume in both vascular structure and flow velocity than the conventional 1-D elevational-focused probe. Compared with the 2-D projection image of the full 3-D SRUS results using the 2-D array probe with the same aperture size, the 2-D projection SRUS image of BEP-SR has similar volume coverage, using 32 folds fewer independent elements. This study demonstrates BEP-SR's ability of high-resolution imaging of microvascular structures and flow velocity within a 3-D volume at significantly reduced costs. The proposed BEP method could significantly benefit the clinical translation of the SRUS imaging technique by making it more affordable and repeatable.

Cartography of Touch: Transformation of Touch through Anatomical Projections

Abstract Both the conception of how people connect changes with the evolution of technology and the circumstances under which we interact changed drastically with the onset of the global COVID-19 pandemic and have changed again during the decline of the pandemic threat. With the art installation Cartography of Touch, the author uses digital and physical media (3D printing and projection mapping of human physiology) to demonstrate the need for human touch. The installation depicts response to touch as the joining of the technological age and the need for essential human interaction through physical touch, which cannot be replaced, using plastic human hands with artificial responses.

3D evaluation of the extracellular matrix of hypoxic pancreatic islets using light sheet fluorescence microscopy

ABSTRACT Pancreatic islet transplantation is a promising treatment for type 1 diabetes, but the survival and function of transplanted islets are hindered by the loss of extracellular matrix (ECM) during islet isolation and by low oxygenation upon implantation. This study aimed to evaluate the impact of hypoxia on ECM using a cutting-edge imaging approach based on tissue clearing and 3D microscopy. Human and rat islets were cultured under normoxic (O2 21%) or hypoxic (O2 1%) conditions. Immunofluorescence staining targeting insulin, glucagon, CA9 (a hypoxia marker), ECM proteins (collagen 4, fibronectin, laminin), and E-cadherin (intercellular adhesion protein) was performed on fixed whole islets. The cleared islets were imaged using Light Sheet Fluorescence Microscopy (LSFM) and digitally analyzed. The volumetric analysis of target proteins did not show significant differences in abundance between the experimental groups. However, 3D projections revealed distinct morphological features that differentiated normoxic and hypoxic islets. Under normoxic conditions, ECM could be found throughout the islets. Hypoxic islets exhibited areas of scattered nuclei and central clusters of ECM proteins, indicating central necrosis. E-cadherin was absent in these areas. Our results, demonstrating a diminution of islets’ functional mass in hypoxia, align with the functional decline observed in transplanted islets experiencing low oxygenation after grafting. This study provides a methodology combining tissue clearing, multiplex immunofluorescence, Light Sheet Fluorescence Microscopy, and digital image analysis to investigate pancreatic islet morphology. This 3D approach allowed us to highlight ECM organizational changes during hypoxia from a morphological perspective.

Concurrent validity and test-retest reliability for the assessment of body segment lengths using two-dimensional video-recordings in weightlifting

ABSTRACT The aims of this study were 1) to propose a simple two-dimensional (2D) video-based method to measure segment lengths in the clean and jerk during weightlifting competitions, and 2) to check concurrent validity and test-retest reliability compared to 3D video-based measurements. In the first part of the study, key body positions for the assessment of segment lengths via 2D measurement method were determined. In the second part, concurrent validity was verified, comparing the 2D segment length measurements to 3D measurements. In the third part, the test-retest reliability of the 2D approach was investigated. In the clean and in the jerk, the lift-off/start position and the position of maximum barbell velocity displayed a minimised offset between ‘real’ 3D segment lengths and 2D projections of the segment lengths. For the concurrent validity assessment, the overall random measurement error was 4.3% with concordance correlation coefficients ranging from 0.64 to 0.88. The standard error of measurement in the test-retest procedure was 2.4% on average with intraclass correlation coefficients ranging from 0.85 to 0.96. The proposed 2D video-based approach to measure body segment lengths during weightlifting exercises is proven to be a valid and reliable procedure.

Cone-beam computed tomography in implant dentistry - guidelines, current concepts and limitations for practice

This article issues scientific background of Cone-beam computed tomography (CBCT) and the importance of taking x-rays before and after implant placement in daily practice as a common care. The review will introduce cone-beam computed tomography guidelines, restrictions and intraoperartive issues for instance nerve damage and bleeding incidents. Modern CBCT technology enables specialists to avoid making a wrong diagnosis, which translates into a higher percentage of people with a positive treatment outcome.
 Diagnostic radiology is a crucial element of every dental treatment planning. CBCT market is expanding gradually since two decades, there are more than 85 distinct CBCT tools available. CBCT is a three-dimensional (3D) imaging used nowadays in dentistry with increased frequency and offers volumetric data on jaw bones and teeth with relatively low radiation doses and costs. Currently, the greatest advantage of CBCT examinations over radiographs is the fact that the image obtained is presented in a 3D projection and not, as is the case, in 2D. It has the ability to help a wider range of patients, but the use of CBCT also has negative consequences. Routine or excessive use has resulted in increased radiation doses accumulating in the patient's body, which translates into an increased risk of adverse effects. The risk varies according to the age of the patient under study and is directly proportional to it, that is, it is highest for young people and lower for older people. The potential risk is also slightly higher in the female population. For this reason, creating awareness of the mandatory patient safety management of CT scans is a key process by which X-ray exposures can be optimised.

Deep-learning-assisted snapshot optical tomography for microscopic volume prediction: a simulation study.

In this simulation study, we demonstrate fast-yet-accurate volume measurement of microscopic objects by combining snapshot optical tomography and deep learning. Snapshot optical tomography simultaneously collects a multitude of projection images and thus can perform 3D imaging in a single snapshot. However, as with other wide-field microscopy techniques, it suffers from the missing-cone problem, which can seriously degrade the quality of 3D reconstruction. We use deep learning to generate a volume prediction from 2D projection images bypassing the 3D reconstruction.

GUPNet++: Geometry Uncertainty Propagation Network for Monocular 3D Object Detection.

Geometry plays a significant role in monocular 3D object detection. It can be used to estimate object depth by using the perspective projection between object's physical size and 2D projection in the image plane, which can introduce mathematical priors into deep models. However, this projection process also introduces error amplification, where the error of the estimated height is amplified and reflected into the projected depth. It leads to unreliable depth inferences and also impairs training stability. To tackle this problem, we propose a novel Geometry Uncertainty Propagation Network (GUPNet++) by modeling geometry projection in a probabilistic manner. This ensures depth predictions are well-bounded and associated with a reasonable uncertainty. The significance of introducing such geometric uncertainty is twofold: (1). It models the uncertainty propagation relationship of the geometry projection during training, improving the stability and efficiency of the end-to-end model learning. (2). It can be derived to a highly reliable confidence to indicate the quality of the 3D detection result, enabling more reliable detection inference. Experiments show that the proposed approach not only obtains (state-of-the-art) SOTA performance in image-based monocular 3D detection but also demonstrates superiority in efficacy with a simplified framework.