Landmark Correspondences Research Articles

Influence of learned landmark correspondences on lung CT registration.

Disease or injury may cause a change in the biomechanical properties of the lungs, which can alter lung function. Image registration can be used to measure lung ventilation and quantify volume change, which can be a useful diagnostic aid. However, lung registration is a challenging problem because of the variation in deformation along the lungs, sliding motion of the lungs along the ribs, and change indensity. Landmark correspondences have been used to make deformable image registration robust to largedisplacements. To tackle the challenging task of intra-patient lung computed tomography (CT) registration, we extend the landmark correspondence prediction model deep convolutional neural network-Match by introducing a soft mask loss term to encourage landmark correspondences in specific regions and avoid the use of a mask during inference. To produce realistic deformations to train the landmark correspondence model, we use data-driven synthetic transformations. We study the influence of these learned landmark correspondences on lung CT registration by integrating them into intensity-based registration as a distance-basedpenalty. Our results on the public thoracic CT dataset COPDgene show that using learned landmark correspondences as a soft constraint can reduce median registration error from approximately 5.46 to 4.08mm compared to standard intensity-based registration, in the absence of lungmasks. We show that using landmark correspondences results in minor improvements in local alignment, while significantly improving globalalignment.

Spatially Resolved Association of Structural Biomarkers on Retinal Function in Non-Exudative Age-Related Macular Degeneration Over 4 Years.

To longitudinally assess the impact of high-risk structural biomarkers for natural disease progression in non-exudative age-related macular degeneration (AMD) on spatially resolved mesopic and scotopic fundus-controlled perimetry testing. Multimodal retinal imaging data and fundus-controlled perimetry stimuli points were semiautomatically registered according to landmark correspondences at each annual visit over a period of up to 4 years. The presence of sub-RPE drusen, subretinal drusenoid deposits, pigment epithelium detachments (PEDs), hyper-reflective foci (HRF), vitelliform lesions, refractile deposits, and incomplete RPE and outer retinal atrophy (iRORA) and complete RPE and outer retinal atrophy (cRORA) were graded at each stimulus position and visit. Localized retinal layer thicknesses were extracted. Mixed-effect models were used for structure-function correlation. Fifty-four eyes of 49 patients with non-exudative AMD (mean age, 70.7 ± 9.1years) and 27 eyes of 27 healthy controls (mean age, 63.4 ± 8.9years) were included. During study course, presence of PED had the highest functional impact with a mean estimated loss of -1.30 dB (P < 0.001) for mesopic and -1.23 dB (P < 0.001) for scotopic testing, followed by HRF with -0.89 dB (mesopic, P = 0.001) and -0.87 dB (scotopic, P = 0.005). Subretinal drusenoid deposits were associated with a stronger visual impairment (mesopic, -0.38 dB; P = 0.128; scotopic, -0.37 dB; P = 0.172) compared with sub-RPE drusen (-0.22 dB, P = 0.0004; -0.18 dB, P = 0.006). With development of c-RORA, scotopic retinal sensitivity further significantly decreased (-2.15 dB; P = 0.02). Thickening of the RPE-drusen-complex and thinning of the outer nuclear layer negatively impacted spatially resolved retinal sensitivity. The presence of PED and HRF had the greatest prognostic impact on progressive point-wise sensitivity losses. Higher predominant rod than cone-mediated localized retinal sensitivity losses with early signs of retinal atrophy development indicate photoreceptor preservation as a potential therapeutic target for future interventional AMD trials.

In-plane Strain Analysis by Correlating Geometry and Visual Data Through a Gradient-Based Surface Reconstruction.

Abnormalities in tissue can be detected and analyzed by evaluating mechanical properties, such as strain and stiffness. While current sensor systems are effective in measuring longitudinal properties perpendicular to the measurement sensor, identifying in-plane deformation remains a significant challenge. To address this issue, this paper presents a novel method for reconstructing in-plane deformation of observed tissue surfaces using a fringe projection sensor specifically designed for measuring tissue deformations. The method employs the latest techniques from computer vision, such as differentiable rendering, to formulate the in-plane reconstruction as a differentiable optimization problem. This enables the use of gradient-based solvers for an efficient and effective optimization of the problem optimum. Depth information and image information are combined using landmark correspondences between the respective image observations of the undeformed and deformed scenes. By comparing the reconstructed pre- and post-deformation geometry, the in-plane deformation can be revealed through the analysis of relative variations between the corresponding models' geometries. The proposed reconstruction pipeline is validated on an experimental setup, and the potential for intraoperative applications is discussed.

Automatic landmark correspondence detection in medical images with an application to deformable image registration.

Deformable image registration (DIR) can benefit from additional guidance using corresponding landmarks in the images. However, the benefits thereof are largely understudied, especially due to the lack of automatic landmark detection methods for three-dimensional (3D) medical images. We present a deep convolutional neural network (DCNN), called DCNN-Match, that learns to predict landmark correspondences in 3D images in a self-supervised manner. We trained DCNN-Match on pairs of computed tomography (CT) scans containing simulated deformations. We explored five variants of DCNN-Match that use different loss functions and assessed their effect on the spatial density of predicted landmarks and the associated matching errors. We also tested DCNN-Match variants in combination with the open-source registration software Elastix to assess the impact of predicted landmarks in providing additional guidance to DIR. We tested our approach on lower abdominal CT scans from cervical cancer patients: 121 pairs containing simulated deformations and 11 pairs demonstrating clinical deformations. The results showed significant improvement in DIR performance when landmark correspondences predicted by DCNN-Match were used in the case of simulated ( ) as well as clinical deformations ( ). We also observed that the spatial density of the automatic landmarks with respect to the underlying deformation affect the extent of improvement in DIR. Finally, DCNN-Match was found to generalize to magnetic resonance imaging scans without requiring retraining, indicating easy applicability to other datasets. DCNN-match learns to predict landmark correspondences in 3D medical images in a self-supervised manner, which can improve DIR performance.

Non-Rigid Point Cloud Matching Based on Invariant Structure for Face Deformation

In this paper, we present a non-rigid point cloud matching method based on an invariant structure for face deformation. Our work is guided by the realistic needs of 3D face reconstruction and re-topology, which critically need support for calculating the correspondence between deformable models. Our paper makes three main contributions: First, we propose an approach to normalize the global structure features of expressive faces using texture space properties, which decreases the variation magnitude of facial landmarks. Second, we make a modification to the traditional shape context descriptor to solve the problem of regional cross-mismatch. Third, we collect a dataset with various expressions. Ablation studies and comparative experiments were conducted to investigate the performance of the above work. In face deformable cases, our method achieved 99.89% accuracy on our homemade face dataset, showing superior performance over some other popular algorithms. In this way, it can help modelers to build digital humans more easily based on the estimated correspondence of facial landmarks, saving a lot of manpower and time.

A Unifying Framework for $n$-Dimensional Quasi-Conformal Mappings

With the advancement of computer technology, there is a surge of interest in effective mapping methods for objects in higher-dimensional spaces. To establish a one-to-one correspondence between objects, higher-dimensional quasi-conformal theory can be utilized for ensuring the bijectivity of the mappings. In addition, it is often desirable for the mappings to satisfy certain prescribed geometric constraints and possess low distortion in conformality or volume. In this work, we develop a unifying framework for computing $n$-dimensional quasi-conformal mappings. More specifically, we propose a variational model that integrates quasi-conformal distortion, volumetric distortion, landmark correspondence, intensity mismatch and volume prior information to handle a large variety of deformation problems. We further prove the existence of a minimizer for the proposed model and devise efficient numerical methods to solve the optimization problem. We demonstrate the effectiveness of the proposed framework using various experiments in two- and three-dimensions, with applications to medical image registration, adaptive remeshing and shape modeling.

3D corrective nose reconstruction from a single image

There is a steadily growing range of applications that can benefit from facial reconstruction techniques, leading to an increasing demand for reconstruction of high-quality 3D face models. While it is an important expressive part of the human face, the nose has received less attention than other expressive regions in the face reconstruction literature. When applying existing reconstruction methods to facial images, the reconstructed nose models are often inconsistent with the desired shape and expression. In this paper, we propose a coarse-to-fine 3D nose reconstruction and correction pipeline to build a nose model from a single image, where 3D and 2D nose curve correspondences are adaptively updated and refined. We first correct the reconstruction result coarsely using constraints of 3D-2D sparse landmark correspondences, and then heuristically update a dense 3D-2D curve correspondence based on the coarsely corrected result. A final refinement step is performed to correct the shape based on the updated 3D-2D dense curve constraints. Experimental results show the advantages of our method for 3D nose reconstruction over existing methods.

Visual navigation with fast landmark selection based on error analysis for asteroid descent stage

Visual navigation is one of the enabling technologies for asteroid landing missions. Quality and efficiency are of paramount importance for determining its robustness and autonomy. This paper proposes a new navigation method, which integrates absolute and relative visual information for quality improvement, and a fast landmark selection method for efficiency purposes. The mapped 3D landmarks with their 2D projections measured in the descent image are primary inputs, and two navigation pipelines, using the extracted 3D-2D and 2D-2D landmark correspondences, are established separately. Then, error analysis, under measurement and system noises, is performed in detail, and based on it, a fast landmark selection algorithm is designed to improve the navigation efficiency. Moreover, the observation model for the attitude motion in adjacent image frames is constructed, followed by the fusion of absolute and relative navigation results using an extended Kalman filter. Finally, Monte Carlo simulations are conducted, and the numerical results demonstrate the advancement of the proposed methods in quality and efficiency.

Landmark-free morphometric analysis of knee osteoarthritis using joint statistical models of bone shape and articular space variability.

Purpose: Osteoarthritis (OA) is a common degenerative disease involving a variety of structural changes in the affected joint. In addition to narrowing of the articular space, recent studies involving statistical shape analysis methods have suggested that specific bone shapes might be associated with the disease. We aim to investigate the feasibility of using the recently introduced framework of functional shapes (Fshape) to extract morphological features of OA that combine shape variability of articular surfaces of the tibia (or femur) together with the changes of the joint space. Approach: Our study uses a dataset of three-dimensional cone-beam CT volumes of 17 knees without OA and 17 knees with OA. Each knee is then represented as an object (Fshape) consisting of a triangulated tibial (or femoral) articular surface and a map of joint space widths (JSWs) measured at the points of this surface (joint space map, JSM). We introduce a generative atlas model to estimate a template (mean) Fshape of the sample population together with template-centered variables that model the transformations from the template to each subject. This approach has two potential advantages compared with other statistical shape modeling methods that have been investigated in knee OA: (i)Fshapes simultaneously consider the variability in bone shape and JSW, and (ii)Fshape atlas estimation is based on a diffeomorphic transformation model of surfaces that does not require a priori landmark correspondences between the subjects. The estimated atlas-to-subject Fshape transformations were used as input to principal component analysis dimensionality reduction combined with a linear support vector machine (SVM) classifier to identify the morphological features of OA. Results: Using tibial articular surface as the shape component of the Fshape, we found leave-one-out cross validation scores of for the classification based on the bone surface transformations alone, for the classification based on the residual JSM, and for the classification using both Fshape components. Similar results were obtained using femoral articular surfaces. The discriminant directions identified in the statistical analysis were associated with medial narrowing of the joint space, steeper intercondylar eminence, and relative deepening of the medial tibial plateau. Conclusions: The proposed approach provides an integrated framework for combined statistical analysis of shape and JSPs. It can successfully extract features correlated to OA that appear consistent with previous studies in the field. Although future large-scale study is necessary to confirm the significance of these findings, our results suggest that the functional shape methodology is a promising new tool for morphological analysis of OA and orthopedics data in general.

Uncertainty-Aware Annotation Protocol to Evaluate Deformable Registration Algorithms.

Landmark correspondences are a widely used type of gold standard in image registration. However, the manual placement of corresponding points is subject to high inter-user variability in the chosen annotated locations and in the interpretation of visual ambiguities. In this paper, we introduce a principled strategy for the construction of a gold standard in deformable registration. Our framework: (i) iteratively suggests the most informative location to annotate next, taking into account its redundancy with previous annotations; (ii) extends traditional pointwise annotations by accounting for the spatial uncertainty of each annotation, which can either be directly specified by the user, or aggregated from pointwise annotations from multiple experts; and (iii) naturally provides a new strategy for the evaluation of deformable registration algorithms. Our approach is validated on four different registration tasks. The experimental results show the efficacy of suggesting annotations according to their informativeness, and an improved capacity to assess the quality of the outputs of registration algorithms. In addition, our approach yields, from sparse annotations only, a dense visualization of the errors made by a registration method. The source code of our approach supporting both 2D and 3D data is publicly available at https://github.com/LoicPeter/evaluation-deformable-registration.

Voting for Distortion Points in Geometric Processing.

Low isometric distortion is often required for mesh parameterizations. A configuration of some vertices, where the distortion is concentrated, provides a way to mitigate isometric distortion, but determining the number and placement of these vertices is non-trivial. We call these vertices distortion points. We present a novel and automatic method to detect distortion points using a voting strategy. Our method integrates two components: candidate generation and candidate voting. Given a closed triangular mesh, we generate candidate distortion points by executing a three-step procedure repeatedly: (1) randomly cut an input to a disk topology; (2) compute a low conformal distortion parameterization; and (3) detect the distortion points. Finally, we count the candidate points and generate the final distortion points by voting. We demonstrate that our algorithm succeeds when employed on various closed meshes with a genus of zero or higher. The distortion points generated by our method are utilized in three applications, including planar parameterization, semi-automatic landmark correspondence, and isotropic remeshing. Compared to other state-of-the-art methods, our method demonstrates stronger practical robustness in distortion point detection.

Non-rigid registration of point clouds using landmarks and stochastic neighbor embedding

Our study presents a probabilistic non-rigid point set registration method to deal with large and uneven deformations. Our method treats the registration as a density estimation problem. In our method, we add two key constraints to enforce landmark correspondences and preserve local neighborhood structure. We assume that the landmarks, which represent the salient points in the point sets, are given or can be detected using keypoint detectors such as scale-invariant feature transform or MeshDOG. By enforcing landmark correspondences, we preserve the overall global shape of the point set with significant deformations. Furthermore, by leveraging stochastic neighbor embedding, we incorporate constraints to preserve local neighborhood structure, which penalizes incoherent transformation within a neighborhood. Our experimental results in both 2D and 3D datasets show that our method outperforms state-of-the-art methods in a large degree of deformations. In particular, quantitative results show that the error is 29% better than the second-best result (from the state-of-the-art methods). Our analysis shows that a relatively small number of landmarks is sufficient to deal with large deformations. Finally, our study shows that our method is computationally comparable to state-of-the-art methods.

Place Recognition in Forests With Urquhart Tessellations

In this letter, we present a novel descriptor based on Urquhart tessellations derived from the position of trees in a forest. We propose a framework that uses these descriptors to detect previously seen observations and landmark correspondences, even with partial overlap and noise. We run loop closure detection experiments in simulation and real-world data map-merging from different flights of an Unmanned Aerial Vehicle (UAV) in a pine tree forest and show that our method outperforms state-of-the-art approaches in accuracy and robustness.

SimLife model: introducing a new teaching device in endocrine surgery simulation

To evaluate the validity and reliability of an innovative training model for endocrine surgical procedures. A simulator training model for endocrine procedures (SimLife) was developed at an academic center. The model consisted of a realistic operating environment with a coherent simulated patient dynamized by pulsatile vascularization with simulated blood warmed to 37 °C, and ventilation. Training sessions were designed for adrenal and thyroid surgery, as well as neck dissection. The primary outcome of interest was to evaluate learners’ performance and satisfaction. Learners’ performance was evaluated based on a scoring scale that followed the Downing method for the assessment of competency. While learners’ satisfaction was evaluated using a Likert scale of 1 to 10 on four items (ease of learning, anatomic correspondence of landmarks, realism, and overall satisfaction). Participants were engaged in 32 training sessions. These included 24 adrenalectomies (conventional and laparoscopic both transabdominal and posterior), and 4 thyroid lobectomies with concomitant functional lateral compartment neck dissection. competency scores were procedure-specific addressing specific core components of a given procedure. Learners’ performance scored above average in all procedures evaluated. Satisfaction scores for the specified four items ranged between 8.43 (SD 0.87) and 8.89 (SD 0.96). No major events were reported for the adrenalectomies, while only one jugular vein injury occurred during neck dissection. SimLife is a hyper-realistic training model that allows for satisfactory acquisition of skills and the evaluation of performance progression. It has the potential to become a cornerstone in specialized surgical training.Electronic supplementary materialThe online version of this article (10.1007/s13304-020-00871-x) contains supplementary material, which is available to authorized users.

ENIGMA

In this paper we propose a fully automatic method for shape correspondence that is widely applicable, and especially effective for non isometric shapes and shapes of different topology. We observe that fully-automatic shape correspondence can be decomposed as a hybrid discrete/continuous optimization problem, and we find the best sparse landmark correspondence, whose sparse-to-dense extension minimizes a local metric distortion. To tackle the combinatorial task of landmark correspondence we use an evolutionary genetic algorithm , where the local distortion of the sparse-to-dense extension is used as the objective function. We design novel geometrically guided genetic operators, which, when combined with our objective, are highly effective for non isometric shape matching. Our method outperforms state of the art methods for automatic shape correspondence both quantitatively and qualitatively on challenging datasets.

Reducing non-realistic deformations in registration using precise and reliable landmark correspondences

Non-rigid image registration is prone to non-realistic deformations. In this paper, we proposed a novel landmark-correspondence detection algorithm, with which, the non-realistic deformations in image registration can be reduced. Our method consists of the following steps. First, the landmarks in the reference image are extracted by a corner detector. Then the landmarks are transferred to the template image by the proposed Multiscale Local Rigid Matching (MsLRM) algorithm. A two-stage method is designed for outlier removal before the landmark correspondences are incorporated into a FFD-based registration through a penalty term considering that the interpolating splines in FFD are highly sensitive to outliers. The proposed method was validated on both simulated images and real-world clinical lung dynamic contrast-enhanced magnetic resonance images. The results showed that the proposed MsLRM achieved sub-pixel accuracy, and was robust to local contrast changes. On clinical datasets, the MsLRM-based landmark-constrained registration improved the registration accuracy by at least 25%, compared with the state-of-the-art registration methods. It achieved an average expert landmark distance of 0.23 mm, close to the inter-observer variability of 0.17 mm. We conclude that our novel landmark-constrained registration improves registration performance on dynamic medical images and outperforms the state-of-the-art registration methods.

Automatic large quantity landmark pairs detection in 4DCT lung images.

To automatically and precisely detect a large quantity of landmark pairs between two lung computed tomography (CT) images to support evaluation of deformable image registration (DIR). We expect that the generated landmark pairs will significantly augment the current lung CT benchmark datasets in both quantity and positional accuracy. A large number of landmark pairs were detected within the lung between the end-exhalation (EE) and end-inhalation (EI) phases of the lung four-dimensional computed tomography (4DCT) datasets. Thousands of landmarks were detected by applying the Harris-Stephens corner detection algorithm on the probability maps of the lung vasculature tree. A parametric image registration method (pTVreg) was used to establish initial landmark correspondence by registering the images at EE and EI phases. A multi-stream pseudo-siamese (MSPS) network was then developed to further improve the landmark pair positional accuracy by directly predicting three-dimensional (3D) shifts to optimally align the landmarks in EE to their counterparts in EI. Positional accuracies of the detected landmark pairs were evaluated using both digital phantoms and publicly available landmark pairs. Dense sets of landmark pairs were detected for 10 4DCT lung datasets, with an average of 1886 landmark pairs per case. The mean and standard deviation of target registration error (TRE) were 0.47±0.45mm with 98% of landmark pairs having a TRE smaller than 2mm for 10 digital phantom cases. Tests using 300 manually labeled landmark pairs in 10 lung 4DCT benchmark datasets (DIRLAB) produced TRE results of 0.73±0.53mm with 97% of landmark pairs having a TRE smaller than 2mm. A new method was developed to automatically and precisely detect a large quantity of landmark pairs between lung CT image pairs. The detected landmark pairs could be used as benchmark datasets for more accurate and informative quantitative evaluation of DIR algorithms.

Inter-Subject Shape Correspondence Computation From Medical Images Without Organ Segmentation

Statistical shape models (SSMs) have been established as robust anatomical priors for medical image segmentation, registration and anatomy modelling. To construct an SSM which accurately models the inter-subject anatomical variations, it is crucial to compute accurate shape correspondence between the training samples. To achieve this goal, the state-of-the-art shape correspondence computation methods always require tedious segmentation of the training images, while they seldom pay enough attention to the correspondence accuracy of key anatomical landmarks like the bone joints, vessel bifurcations, etc. As a result, the computation of shape correspondence is time-consuming and the correspondence accuracy is imperfect. To solve these problems, this paper proposes a novel shape correspondence computation approach which eliminates the need for image segmentation by registering an organ shape template to the training images. This method allows the human expert to specify key anatomical landmarks in the training images to define the correspondence of the crucial landmarks. An intensity-and-landmark-combined strategy is implemented to utilized both the image intensity and expert landmarks to obtain accurate shape correspondence. This method is evaluated for the construction of head anatomy SSM and spine SSM based on computed tomography (CT) images. The SSMs constructed using the proposed method demonstrates better shape correspondence accuracy than other state-of-the-arts correspondence methods. In particular, this method obtains pixel-level surface correspondence accuracy (1.38 mm) for the skull and sub-pixel level accuracy (0.92 mm) for the spine. The generalisability and specificity of the SSMs constructed using our method are also superior to SSMs constructed using other compared correspondence methods. With this method, we propose a novel approach which takes less human intervention and produces higher quality SSM with better shape modelling accuracy.

Technical Note: A method for quality assurance of landmark sets for use in evaluation of deformable image registration accuracy of lung parenchyma.

To develop a quality control method to improve the accuracy of corresponding landmark sets used for deformable image registration (DIR) evaluation in the lung parenchyma. An iterative workflow was developed as a method for quality assurance of landmark sets. Starting with the initial landmark set for a given image pair, a landmark-based deformation was applied to one of the images. A difference image and a color overlay were generated using the deformed image and the other image of the pair. Inspection of these generated images at locations of landmarks allowed for the identification of misplaced landmarks. The observer responsible for creating the initial landmark set was tasked with review and revision of points flagged by the quality assurance procedure. Using the updated landmark sets, the process was repeated until all points were acceptable to the reviewer. Eighteen landmark sets, containing a mean (SD) of 170 (31) landmarks, were created using CT images from non-small cell lung cancer patients exhibiting large geometric changes and atelectasis resolution, making landmark specification challenging. Following the quality assurance procedure, the final landmark sets contained a mean (SD) of 165 (25) landmarks, as points too difficult to match were removed and points were added to regions deficient in landmarks. For landmark sets in which changes were made, maximum and mean differences in landmark positions before and after quality assurance ranged between 8.7-81.5mm and 0.3-9.6mm, respectively. An effective method for improving the accuracy of landmark correspondence was presented. This quality assurance approach enables more accurate evaluation of DIR for lung parenchyma in clinical image pairs in the absence of a ground truth deformation and may be applicable to other feature-rich anatomical sites.

Continuous and orientation-preserving correspondences via functional maps

We propose a method for efficiently computing orientation-preserving and approximately continuous correspondences between non-rigid shapes, using the functional maps framework. We first show how orientation preservation can be formulated directly in the functional (spectral) domain without using landmark or region correspondences and without relying on external symmetry information. This allows us to obtain functional maps that promote orientation preservation, even when using descriptors, that are invariant to orientation changes. We then show how higher quality, approximately continuous and bijective pointwise correspondences can be obtained from initial functional maps by introducing a novel refinement technique that aims to simultaneously improve the maps both in the spectral and spatial domains. This leads to a general pipeline for computing correspondences between shapes that results in high-quality maps, while admitting an efficient optimization scheme. We show through extensive evaluation that our approach improves upon state-of-the-art results on challenging isometric and non-isometric correspondence benchmarks according to both measures of continuity and coverage as well as producing semantically meaningful correspondences as measured by the distance to ground truth maps.