This paper presents a shape anchor guided learning strategy (AncLearn) for robust holistic indoor scene understanding. We observe that the search space constructed by current methods for proposal feature grouping and instance point sampling often introduces massive noise to instance parsing and object reconstruction. Accordingly, we develop AncLearn to learn the shape anchors of 3D objects, to provide shape constraints in a top-down manner. The learned anchors fit to instance surfaces and thus provide fine localization clues for (i) separating noise and object-related features to offer reliable instance proposals, (ii) reducing outliers in object point sampling for providing well-structured geometry priors for object reconstruction and (iii) integrating RGB information at the instance level to incorporate additional texture clues for better semantic perception. We embed AncLearn strategy into a reconstruction-from detection learning system (AncRec++) to generate high-quality semantic scene models in a purely instance-oriented manner. Within AncLearn, the network learns to abstract the shape priors of each instance and thus improves its robustness against the sparsity and incompleteness of point clouds. Experiments conducted on the challenging ScanNetv2 dataset demonstrate that our method consistently achieves state-of-the-art performance in terms of 3D object detection, layout estimation, and shape reconstruction. Code is available at https://github.com/Geo-Tell/AncRec.

The aim is to develop and test a framework that integrates real-time catheter shape reconstruction, interactive simulations, and mixed reality visualization to enable accurate monitoring of catheter vessel interactions during endovascular navigation. A finite element model (FEM) of the venous pathway from the right femoral vein to the inferior vena cava was generated from computed tomography data and implemented into an interactive simulation. Catheter motion was imposed as boundary condition, and catheter vessel contact was modeled with a Lagrange multiplier formulation to compute vessel deformation. The framework was integrated and tested in vitro using a sensorized catheter with fiber Bragg grating and electromagnetic sensors to provide real-time 3D shape and location as it was advanced by a catheter driver through a silicone replica of the vascular anatomy. Upon registration, real-time sensor read-outs fed the simulation, and the updated catheter and vessel geometries were streamed to Hololens (HL2). The performance of the simulation and the accuracy of FEM-computed vessel wall displacement were validated vs. experimental ground truth obtained via stereo frames triangulation. Complexity and extent of catheter vessel interaction affected FEM performance by increasing the computational cost. The simulated time exceeded the real temporal extent of the physical phenomenon by 12% during the initial navigation phase and by 45% when the catheter reached the most tortuous portion of the vessel. The HL2 rendering remained stable between 35 and 40 frames per second. Across these two phases, the median relative displacement error between FEM-computed vessel wall displacements and the ground truth remained below 1mm and 2.33 mm, respectively. The study demonstrates the feasibility of integrating interactive biomechanical simulation with real-time sensor data to enable continuous monitoring of catheter vessel interactions, with mixed reality visualization serving as a user interface to support a more engaged and better-informed operator throughout the navigation.

In this Letter, a line-scan phase measuring deflectometry (LSPMD) method for online three-dimensional (3D) shape reconstruction of specular surfaces is proposed for the first time, to the best of our knowledge. Differing from conventional PMD systems, LSPMD exploits a line-scan camera and a high-frame-rate sinusoidal fringe display screen to realize an ultrafast fringe acquisition rate and sufficient resolution. Furthermore, based on the novel hardware architecture and acquisition logic, a flexible and robust system calibration method for LSPMD using a geometrically defined surface is specifically proposed, which enables high-precision 3D reconstruction without camera calibration and additional auxiliary patterns. The experimental results illustrate that LSPMD simultaneously achieves an ultra-high acquisition rate of 10 kHz and a high pixel resolution of 8132 pixels.

Insects navigate cluttered environments using slender, flexible antennae densely packed with mechanosensors, a lightweight, energy-efficient solution for tactile perception. We introduce CITRAS (Cockroach-Inspired Tactile Robotic Antenna Sensor), a miniature, compliant, multi-segment tactile probe aimed at enabling similarly capable close-range perception on insect-scale robots under stringent size, mass, and power constraints. CITRAS (total size:mm; mass: ) features eight flexural hinge segments, each with high-resolution capacitive sensors embedded within a compliant multilayer laminate structure, that detect femtofarad-scale capacitance changes induced by hinge deflection. Through systematic mechanical and sensing characterization under both quasi-static and dynamic conditions, we demonstrate sub-degree angular precision (max error ∘), accurate shape reconstruction, and consistent repeatable performance with minimal hysteresis in slow bending. Under rapid interactions, CITRAS exhibits low damping and rich dynamic responses that encode environmental features. We further validate the system in three core tactile tasks: estimating body-to-wall distance (error ), measuring object gap width (error ), and discriminating between smooth and rough surface textures via spatiotemporal tactile images. These results show that CITRAS delivers a compact, distributed, bioinspired tactile modality capable of reliable environment sensing, filling a critical gap in perception for insect-scale robots. Furthermore, the antenna consumes only (excluding MCU), making it suitable for future full deployment onboard insect-scale robots and thus paves the way for autonomous navigation and interaction in confined, unstructured, or delicate environments at this scale.

Accurate reconstruction of cardiac anatomy from sparse clinical images remains a major challenge in patient-specific modeling. While neural implicit functions have previously been applied to this task, their application to mapping anatomical consistency across subjects has been limited. In this work, we introduce Neural Implicit Heart Coordinates (NIHCs), a standardized implicit coordinate system, based on universal ventricular coordinates, that provides a common anatomical reference frame for the human heart. Our method predicts NIHCs directly from a limited number of 2D segmentations (sparse acquisition) and subsequently decodes them into dense 3D segmentations and high-resolution meshes at arbitrary output resolution. Trained on a large dataset of 5000 cardiac meshes, the model achieves high reconstruction accuracy on clinical contours, with mean Euclidean surface errors of 2.51 ± 0.33 mm in a diseased cohort (n=4549) and 2.31 ± 0.36 mm in a healthy cohort (n=5576). The NIHC representation enables anatomically coherent reconstruction even under severe slice sparsity and segmentation noise, faithfully recovering complex structures such as the valve planes. Compared with traditional pipelines, inference time is reduced from over 60 s to 5-15 s. These results demonstrate that NIHCs constitute a robust and efficient anatomical representation for patient-specific 3D cardiac reconstruction from minimal input data. The code is available at https://github.com/marsof97/NIHC.

Human fingers are one of the most remarkable organs for handling complex tasks or manipulating unknown objects, not only due to its dexterous and powerful movement capabilities but also its rich kinematic sense at joints and tactile sensing at skins. Shape and tactile sensing are crucial for soft pneumatic fingers to achieve embodied intelligence. Reliable tactile sensing of soft pneumatic-driven robots is particularly challenging due to its large deformation and adaptability. Here, we propose a distributed local curvature sensing-based solution for simultaneous shape and tactile perceptions in real-time. Utilizing 4 seamlessly integrated bidirectional bending curvature sensing units, real-time shape curve, contact location, and contact force can be obtained. Experimental results indicate a maximum shape reconstruction error of 0.3 mm (when the reconstruction length is 90 mm) and a force estimation error of 0.02 N (RMSE, range 0-0.4 N). Moreover, a two-finger gripper was developed; shape and tactile sensing during grasping of diverse objects (varies in weight, size, stiffness) and force-controlled grasping are achieved. Utilizing the shape-sensing and contact-event detection capabilities, dimension of the grasped objects can be recognized in real-time. This work provides an effective, highly robust, easy-to-implement, and transformative perception solution for soft bionic fingers and beyond.

3D shape reconstruction is an active area of research and a fundamental problem in computer vision, with growing applications in the medical domain, where it enables the recovery of missing or fine anatomical structures. Numerous approaches have been proposed for medical shape reconstruction, emphasizing accurate and anatomically plausible reconstructions. However, 3D shape reconstruction remains an inherently ill-posed problem, meaning (1) both neural network-based methods and conventional shape modeling approaches naturally introduce uncertainty in their predictions, and (2) multiple anatomically plausible reconstructions exist for a given partial or low-resolution input. While the uncertainty aspects have been widely explored in general computer vision, they remain relatively under-explored in the context of medical shape reconstruction. In this paper, we developed a 3D Bayesian U-Net and investigated its use for uncertainty estimation and probabilistic reconstructions across three key tasks: cranial reconstruction, facial reconstruction, and skull shape super-resolution. Our findings show that the Bayesian model is able to produce a range of anatomically plausible skull reconstructions while capturing natural skull variations arising from the learned weight uncertainty. Notably, these variations are primarily expressed through differences in bone thickness, which aligns with anatomical expectations, particularly relevant in real-world applications like cranial implant design. Additionally, we propose a principled framework to study the relationship between weight uncertainty and reconstruction uncertainty by analyzing the learned posterior distribution of the weights, demonstrating that our Bayesian U-Net achieves comparable reconstruction performance to a deterministic U-Net baseline while providing reliable uncertainty estimates. Our study also reveals a clear cross-task uncertainty pattern, where tasks with stronger structural constraints, like super-resolution, yield lower predictive uncertainty, while less constrained tasks, like facial reconstruction, result in higher uncertainty. Refer to the project page for more visual results https://git.zib.de/jli/uncertainty-aware-skull-reconstruction/.

A computationally efficient finite element method for shape reconstruction of inverse conductivity problems

Internal anatomical motion challenges precise radiation delivery during external beam radiotherapy. Estimating and compensating for anatomical motion are essential for improving planned dose delivery to target volumes while sparing organs-at-risk. This research achieves accurate motion prediction using only planar X-ray imaging from conventional linear accelerators, without surrogate signals or invasive fiducial markers. We propose Deep-Motion-Net: a patient-specific end-to-end graph neural network (GNN) enabling 3D volumetric organ reconstruction from single in-treatment kV planar X-ray images at arbitrary projection angles. A 2D convolutional neural network (CNN) encoder extracts image features, which four feature pooling networks fuse to a 3D template organ mesh. A ResNet-based graph attention network then deforms the feature-encoded mesh. Training uses synthetically generated organ motion instances and corresponding kV images, created by deforming a reference CT volume aligned with the template mesh, generating digitally reconstructed radiographs (DRRs) at required angles, and applying DRR-to-kV style transfer via conditional CycleGAN. Quantitative testing on synthetic respiratory motion scenarios and qualitative assessment on in-treatment images from four liver cancer patients demonstrated overall mean prediction errors of 0.16±0.13 mm, 0.18±0.19 mm, 0.22±0.34 mm, and 0.12±0.11 mm across datasets. Mean peak prediction errors were 1.39 mm, 1.99 mm, 3.29 mm, and 1.16 mm. This approach leverages accessible in-treatment imaging, avoiding expensive MRI systems or invasive markers. To the best of our knowledge, this is the first deep learning framework reconstructing volumetric 3D organ models from single-view images at arbitrary angles throughout an entire in-treatment scan series. Our approach achieves sub-millimetre accuracy when validated on synthetic motion instances and demonstrates clinical feasibility on real-treatment kV images, for which volumetric ground truth is inherently unavailable. The code is available at https://github.com/isurusuranga/DeepMotionNet .

DFPPNet: 3D shape reconstruction of dynamic object via FPP method without motion-induced error

Abstract The acoustic inverse scattering problem is of critical importance in a number of fields, including medical imaging, sonar, and non-destructive evaluation. The problem of interest can vary from the detection of the shape to the properties of an obstacle. The challenge is that this problem is severely ill-posed and highly nonlinear. Significant efforts have been expended over the years to develop solutions to this problem. However, existing fast data-driven methods primarily focus on the two-dimensional scattering case. This paper explores the potential of using machine learning to accelerate the solution to the three-dimensional version of the problem. To this end, we develop ISSRNet, a deep learning framework for 3D shape reconstruction using phaseless far-field data. The framework is implemented by (a) using a compact probabilistic shape latent space learned by a 3D variational auto-encoder, and (b) a convolutional neural network trained to extract far-field features due to multiple incident waves and map the acoustic scattering information to this shape representation. We demonstrate ISSRNet's 3D shape reconstruction capabilities on random rock-like particles, and airplane objects from the popular ShapeNet data set. We also evaluate the framework's performance when trained on lower-resolution scattering data and when receiver locations include uncertainty. Our experiments show that the proposed framework is able to capture both global and local details, differentiate between different types of shapes and performs several orders of magnitude faster than its numerical iterative counterparts.

This Letter presents a novel, to our knowledge, multi-scale EKF-based phase shift estimation for motion-induced compensation. A phase state space model is constructed using fringe information, and an EKF phase shift estimator is established based on the Bernoulli distribution. A multi-scale sliding window is designed to traverse the full-field fringe pattern, and the Bernoulli distribution-based guidance yields enhanced estimation accuracy at the edge of the measured object. The actual phase shifts are evaluated to effectively reduce the motion-induced phase error. The experimental results of the proposed method demonstrate the reliability in the rotation and translation motion and achieve accurate 3D shape reconstruction in dynamic scenes.

Looping during colonoscopy procedures is a major cause of patient discomfort and increases the risk of complications. With the rising incidence of colorectal cancer, there is an urgent need for safer and more efficient colonoscopy techniques. We developed a proof-of-concept platform for 3D visualization of the colonoscope combined with artificial intelligence (AI)-based loop detection. The system consists of an array of 15 inertial measurement units (IMUs) mounted on a flexible printed circuit board that can be retrofitted into the instrument channel of conventional colonoscopes. Using sensor fusion algorithms, the position and orientation of individual IMUs are estimated to reconstruct the 3D shape of the colonoscope in real time. The system successfully reconstructed the colonoscope shape in a silicone colon phantom, and the AI model achieved high loop-detection performance with an area under the receiver operating characteristic curve of 0.95 on test data. These results demonstrate feasibility in an artificial environment, while further validation in clinical settings is required.

In underwater scattering environments, the attenuation effect of light makes it challenging to capture clear images and realize the 3D reconstruction of underwater objects. Single-pixel imaging (SPI) can overcome the limitations of traditional optical imaging, but its imaging efficiency is low due to the cumbersome acquisition process. This paper proposes a new, to our knowledge, Fourier SPI method based on hNet to achieve phase unwrapping and 3D reconstruction of underwater objects at low sampling rates under high-turbidity conditions. First, a 3D-UFSPI-hNet network is designed to accurately recover the absolute phases of objects, and it can reconstruct highly accurate absolute phases from low-sampling-rate deformed fringe patterns in strongly scattering environments, thereby reducing the time required for image acquisition and phase unwrapping. Subsequently, the 3D shape reconstruction of underwater objects is achieved by combining the system calibration parameters using the self-developed SPI system. Numerical simulations and physical experiments show that the proposed method can achieve high-quality absolute phase reconstruction images and 3D imaging at a sampling rate of 5% and a turbidity of 50NTU for the first time. Compared with 3D-SPI methods based on traditional Fourier SPI, such as the UNet and the UNet 3+, our approach offers a new strategy for underwater optical 3D imaging, especially in environments with low sampling rates and strong scattering conditions.

Feasibility of whole heart 3D shape reconstruction from sparse echocardiographic views using CT-derived simulations.

Reliable three-dimensional shape feedback is essential for magnetic soft continuum robots (MSCRs) to navigate safely within highly confined and tortuous vascular pathways. Fiber Bragg grating (FBG) sensors provide a promising solution to this issue, due to their compact form, high sensitivity, and immunity to electromagnetic interference. However, the low elastic modulus of MSCRs leads to substantial strain-transfer loss between the robot body and the embedded FBGs. Meanwhile, the large-curvature bending and torsional deformation introduce significant errors in curvature and bending-direction estimation. To overcome these challenges, this work introduces a compact and cost-effective shape-sensing scheme by using sparse FBG arrays with an optimized curvature/angle estimation algorithm, to realize robust 3D shape reconstruction. A curvature correction coefficient and a twist-angle offset are used to compensate for material-dependent transfer and alignment errors, while a curvature-pair interpolation algorithm, tailored for C<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> continuous spatial curves, enables accurate curvature/angle estimation at non-sensing locations and improves the MSCR’s shape reconstruction accuracy. Experimental calibration and validation demonstrate that the proposed scheme can achieve submillimeter-level shape reconstruction and millimeter-scale tip positioning precision, satisfying the clinical requirements for minimally invasive interventions.

Understanding the interplay between crystalline geometry and refractive error is important to get insights into lens aging and guide the design and selection of artificial intraocular lenses that replace the aged crystalline lens in refractive lens exchange and cataract surgery. In this study we evaluated the relationships between the full geometry of the crystalline lens and age, axial length (AL), and refractive error in patients scheduled for cataract surgery. Optical Coherence Tomography (IOLMaster700, Zeiss) combined with custom-developed full crystalline lens shape reconstruction algorithms were applied to image 453 eyes from 327 subjects and quantify crystalline lens geometrical parameters, including thickness (LT), radius of the anterior and posterior surface (RAL, RPL), volume (LV), surface area (LSA), diameter (DIA) and equatorial plane position (EPP). Correlation and partial correlation analysis (controlling for age, AL, and refraction) were performed. Multiple linear regression models evaluated whether age and AL or refraction predicted LT, DIA, LV and LSA. Age was significantly correlated with LT, LV, LSA, DIA, EPP, and refraction, and these associations persisted after controlling for AL or manifest refraction spherical equivalent. AL was correlated with LT, EPP, DIA, RAL, and with DIA, RAL, RPL and LSA after adjusting for age. Overall, the lens expands axially and meridionally with age, increasing in thickness, diameter, volume and surface area, accompanied by a hyperopic shift. Longer eyes exhibited lenses with larger diameters and flatter anterior and posterior surfaces, partially offsetting myopia. These findings suggest that age- and growth-related lens remodeling contributes to refractive shifts.

We introduce Motion2VecSets, a 4D diffusion model for dynamic surface mesh generation from various ambiguous observations, including a sequence of RGB images, sparse and partial point clouds, and low-resolution voxel grids. While recent methods using neural field representations have shown success in modeling non-rigid objects, conventional feed-forward architectures struggle with noisy, partial, or sparse observations due to their deterministic nature. To address the inherent one-to-many mapping problem, we introduce a diffusion model that explicitly learns the shape and motion distribution of non-rigid objects through an iterative denoising process of compressed latent representations. The diffusion-based priors provide more plausible and diverse reconstructions under ambiguous conditions. Instead of relying on global latent codes, we represent 4D dynamics using latent sets. This novel 4D representation captures local shape and deformation patterns, leading to more accurate non-linear motion capture and significantly improving generalization capacity to unseen motions and identities. For temporally coherent tracking, we jointly denoise latent sets across frames and enable cross-frame information exchange. To reduce computational cost, we design an interleaved spatial-temporal attention block that alternately aggregates deformation latents along spatial and temporal dimensions. Extensive experiments on datasets of humans, animals, and articulated objects demonstrate that Motion2VecSets outperforms prior methods in reconstructing and tracking non-rigid deformations from various imperfect observations. Our implementation is available at https://vveicao.github.io/projects/Motion2VecSets/.

During percutaneous coronary intervention, conventional 2D X-ray imaging lacks depth information, making it difficult for clinicians to determine the 3D position of the guidewire. While some recent approaches incorporate micro-sensors to assist with pose estimation, many rely on implanted electromagnetic sensors, which can introduce additional clinical risks. In the paper, we present a non-invasive alternative by using an external 3-axis electronic magnetometer array. We further propose a Local-Global Magneto-Visual Network framework (LG-MagNet) that fuses magnetic field information with image data to enable precise 3D pose estimation of the guidewire. Specifically, we first perform a shared encoder for cross-modal feature fusion. Then we employ convolutional operations that integrate local and global features. Finally, we utilize a lightweight prediction head for end-to-end depth regression. We constructed experimental equipment and collected a clinical simulation datasets. Results show a root mean square error (RMSE) of (0.797 ± 0.095 mm) for depth prediction along the Z-axis and an overall RMSE of (1.216 ± 0.072) mm for 3D guidewire shape reconstruction. Quantitative analysis indicates that fusing external magnetometer data with 2D imaging improves pose estimation stability, particularly in regions with curvature.

Adaptive 3D shape reconstruction using multi-exposure fusion and gradient curve