Thin Plate Spline Transformation Research Articles

Continuous Piecewise-Affine Based Motion Model for Image Animation

Image animation aims to bring static images to life according to driving videos and create engaging visual content that can be used for various purposes such as animation, entertainment, and education. Recent unsupervised methods utilize affine and thin-plate spline transformations based on keypoints to transfer the motion in driving frames to the source image. However, limited by the expressive power of the transformations used, these methods always produce poor results when the gap between the motion in the driving frame and the source image is large. To address this issue, we propose to model motion from the source image to the driving frame in highly-expressive diffeomorphism spaces. Firstly, we introduce Continuous Piecewise-Affine based (CPAB) transformation to model the motion and present a well-designed inference algorithm to generate CPAB transformation from control keypoints. Secondly, we propose a SAM-guided keypoint semantic loss to further constrain the keypoint extraction process and improve the semantic consistency between the corresponding keypoints on the source and driving images. Finally, we design a structure alignment loss to align the structure-related features extracted from driving and generated images, thus helping the generator generate results that are more consistent with the driving action. Extensive experiments on four datasets demonstrate the effectiveness of our method against state-of-the-art competitors quantitatively and qualitatively. Code will be publicly available at: https://github.com/DevilPG/AAAI2024-CPABMM.

UTTSR: A Novel Non-Structured Text Table Recognition Model Powered by Deep Learning Technology

To prevent the compilation of documents, many table documents are formatted with non-editable and non-structured texts such as PDFs or images. Quickly recognizing the contents of tables is still a challenge due to factors such as irregular formats, uneven text quality, and complex and diverse table content. This article proposes the UTTSR table recognition model, which consists of four parts: text region detection, text line detection and recognition, and table sequence recognition. For table detection, the Cascade Faster RCNN with the ResNeXt105 network is implemented, using TPS (Thin Plate Spline) transformation and affine transformation to correct the image and to improve accuracy. For text line detection, DBNET is used with Do-Conv in FPN (Feature Pyramid Networks) to speed up training. Text lines are recognized using CRNN without the CTC module, enhancing recognition performance. Table sequence recognition is based on the transformer combined with post-processing algorithms that fuse table structure sequences and unit grid content. Experimental results show that the UTTSR model outperforms the compared methods. This upgraded model significantly improves the accuracy of the previous state-of-the-art F1 score on complex tables, reaching 97.8%.

Cross Sectional Analysis of Eurasian Skull Anatomy for 3D Cephalometry-Normative Data Reveal Four Different Skull Types.

The unsolved problem in three-dimensional surgical planning for patients with facial deformity, dysgnathia, or asymmetry is the lack of a normative database of "norm skulls" that can be used as treatment objectives. A study was conducted on 90 Eurasian persons (46 male and 44 female adults) for whom cone beam-computed tomography images were available. Inclusion criteria were adult patients with a skeletal Class I pattern, proper interincisal relationship with normal occlusion, the absence of an open bite both in the anterior and posterior region, and a normal and balanced facial appearance; patients with dysgnathia and malformations were excluded. A total of 18 landmarks were digitized and 3D cephalometric measurements were performed and analyzed by means of proportions calculated from the landmarks. Male and female skulls were analyzed, as well as subdivisions revealed by cluster analysis. The data showed that four subtypes of skulls were distinguishable with statistical significance (p < 0.05). A male and a female type subdivided in a brachiocephalic and dolichocephalic phenotype could be identified. For each type, a mean shape was calculated by a Procrustes transformation, which, in turn, was used to create four template skulls from a male and a female skull. This was accomplished by fitting the polygon models of the two skulls to each of the two subtypes based on the landmarks marked on them using a thin plate spline transformation. The normative data of the subtypes can individually serve as a guide for orthodontic surgery in the Eurasian population, which is especially helpful in 3D planning and the execution of craniofacial operations.

VCFN: Virtual Cloth Fitting Try-On Network

Virtual cloth fitting network has an increasing demand with a growing online shopping trend to map target clothes on reference subject. Previous research depicts limitations in the generation of promising deformed clothes on the wearer's body while retaining the design features of cloth-like logo, text, and wrinkles. The proposed model first learns thin-plate spline transformations to warp images according to body shape, followed by a try-on module. The former model combines deformed cloth with a rendered image to generate a composition mask and outputs target body without blurry clothes while preserving critical requirements of the wearer. Experiments are performed on the Zalando dataset and the model produces fine richer details and promised generalized results.

FontRL: Chinese Font Synthesis via Deep Reinforcement Learning

Automatic generation of Chinese fonts is a valuable but challenging task in areas of AI and Computer Graphics, mainly due to the huge amount of Chinese characters and their complex glyph structures. In this paper, we propose FontRL, a novel method for Chinese font synthesis by using deep reinforcement learning. Specifically, we first train a deep reinforcement learning model to obtain the Thin-Plate Spline (TPS) transformation that is able to modify the reference stroke skeleton in a mean font style into the skeleton of a required style for each stroke of every unseen Chinese character. Afterwards, we utilize a CNN model to predict the location and scale information of these strokes, and then assemble them to get the skeleton of the corresponding character. Finally, we convert each synthesized character skeleton into the glyph image via an image-to-image translation model. Both quantitative and qualitative experimental results demonstrate the superiority of the proposed FontRL compared to the state of the art. Our code is available at https://github.com/lsflyt-pku/FontRL.

Artificial intelligence and CT-based 3D statistical modeling to assess transsacral corridors and plan implant positioning.

Transsacral corridors at levels S1 and S2 represent complex osseous spaces allowing percutaneous fixation of non- or minimally-displaced fragility fractures of the sacrum. To safely place transsacral implants, they must be completely intraosseous. However, standard radiographs and CT do not properly demonstrate the corridor's intricate configuration. Our goal was to facilitate the three-dimensional assessment of transsacral corridors using artificial intelligence and the planning of transsacral implant positioning. In total, 100 pelvic CTs (49 women, mean age: 58.6 ± SD 14.8 years; 51 men, mean age: 60.7 ± SD 13 years) were used to compute a 3D statistical model of the pelvic ring. On the basis of morphologic features (=predictors) and principal components scores (=response), regression learners were interactively trained, validated, and tuned to predict/sample personalized 3D pelvic models. They were matched via thin-plate spline transformation to a series of 20 pelvic CTs with fragility fractures of the sacrum (18 womenand 2 men, age: 69-9.5 years, mean age: 78.65 ± SD 8.4 years). These models demonstrated the availability, dimension, cross-section, and symmetry of transsacral corridors S1 and S2,as well as the planned implant position, dimension, axes, and entryand exit points.The complete intraosseous pathway was controlled in CT reconstructions. We succeeded to establish a workflow determining transsacral corridors S1 and S2 using artificial intelligence and 3D statistical modeling.

Flexible scene text recognition based on dual attention mechanism

SummaryScene text recognition (STR) is a very popular topic in the field of computer vision, which can extract text from complex natural scenes. In this article, we propose an end‐to‐end trainable and flexible STR method based on a dual attention mechanism. The proposed method consists of four modules: a thin plate spline transformer for normalizing the original image, a Channel‐Att feature extractor for obtaining representative features, a bidirectional long short‐term memory encoder for encoding sequential context features, and a Self‐Att based decoder for predicting text labels. The results on seven different benchmark datasets IIIT, SVT, IC03, IC13, IC15, SVTP, and CUTE, show that the proposed method is comparable to 13 existing methods. Especially, the average text recognition accuracy of the proposed method is about 1.4% higher than the state‐of‐the‐art method.

Automatic craniofacial registration based on radial curves

Accurate registration of three-dimensional craniofacial surface is a foundation for craniofacial reconstruction. Most of craniofacial registration methods need to manually calibrate the feature points. This paper presents an automatic craniofacial registration method based on radial curves. First, we extract radial curves on both the reference and target craniofacial surfaces, and suggest an automatic feature point selection algorithms on these radial curves. Second, we align the two sets of radial curves, and develop a consistency point drift (CPD) algorithm to establish correspondences between feature points on the reference and target surfaces. Then, these feature points are used as control points of the thin plate spline transformation (TPS) algorithm, and the reference craniofacial surface is transformed toward the target craniofacial surface. Finally, the corresponding points of the reference surface on the target surface are searched to realize the registration. The experimental results show that the proposed algorithm is effective.

Convolutional Neural Network Architecture for Geometric Matching.

We address the problem of determining correspondences between two images in agreement with a geometric model such as an affine, homography or thin-plate spline transformation, and estimating its parameters. The contributions of this work are three-fold. First, we propose a convolutional neural network architecture for geometric matching. The architecture is based on three main components that mimic the standard steps of feature extraction, matching and simultaneous inlier detection and model parameter estimation, while being trainable end-to-end. Second, we demonstrate that the network parameters can be trained from synthetically generated imagery without the need for manual annotation and that our matching layer significantly increases generalization capabilities to never seen before images. Finally, we show that the same model can perform both instance-level and category-level matching giving state-of-the-art results on the challenging PF, TSS and Caltech-101 datasets.

ASTER: An Attentional Scene Text Recognizer with Flexible Rectification.

A challenging aspect of scene text recognition is to handle text with distortions or irregular layout. In particular, perspective text and curved text are common in natural scenes and are difficult to recognize. In this work, we introduce ASTER, an end-to-end neural network model that comprises a rectification network and a recognition network. The rectification network adaptively transforms an input image into a new one, rectifying the text in it. It is powered by a flexible Thin-Plate Spline transformation which handles a variety of text irregularities and is trained without human annotations. The recognition network is an attentional sequence-to-sequence model that predicts a character sequence directly from the rectified image. The whole model is trained end to end, requiring only images and their groundtruth text. Through extensive experiments, we verify the effectiveness of the rectification and demonstrate the state-of-the-art recognition performance of ASTER. Furthermore, we demonstrate that ASTER is a powerful component in end-to-end recognition systems, for its ability to enhance the detector.

Offline signature verification using shape correspondence

Biometrics has always been an integral part of human identification and verification, with offline signature verification being a most crucial component of it. It is a challenging task as the signatures are time variant. To address the above difficulty, this paper presents a novel approach to identify the correspondence between pixels of different signatures using an adaptive weighted combination of shape context distance and Euclidean distance. These correspondences are then used for the transformation of query signature plane to reference signature plane using thin plate spline transformation. The distances between signatures are computed using plane transformation, a shape descriptor, and the farness between matched pixels. The computed distances are then fed to the support vector machine (SVM) classifier to determine the merit of genuineness. With the proposed methodology, better accuracy is obtained. The results exhibit an accuracy of 89.58% using proposed method on GPDS synthetic signature database.

Offline signature verification using shape correspondence

Biometrics has always been an integral part of human identification and verification, with offline signature verification being a most crucial component of it. It is a challenging task as the signatures are time variant. To address the above difficulty, this paper presents a novel approach to identify the correspondence between pixels of different signatures using an adaptive weighted combination of shape context distance and Euclidean distance. These correspondences are then used for the transformation of query signature plane to reference signature plane using thin plate spline transformation. The distances between signatures are computed using plane transformation, a shape descriptor, and the farness between matched pixels. The computed distances are then fed to the support vector machine (SVM) classifier to determine the merit of genuineness. With the proposed methodology, better accuracy is obtained. The results exhibit an accuracy of 89.58% using proposed method on GPDS synthetic signature database.

A proposal for touching component segmentation in Arabic manuscripts

Text-line segmentation is one of the key factors which affect the performance of handwriting recognition system. Therefore, to make recognition systems more effective and accurate, segmentation of touching text-lines is an important task. One of the problems making this task crucial is the presence of touching components (TCs) representing connections between word letters of consecutive text-lines or those of words of the same text-line. The proposed method aims to segment TCs. It is mainly based on two steps: (1) finding for a localized TC a similar model, stored in a dictionary with its correct segmentation, using shape context descriptor and an interpolation function: the thin plate spline transformation, (2) segmenting the TC based on central point of the found similar model parts. TCs are assumed to be already extracted from Arabic manuscript images. Experiments are carried on a common TC database, using two metrics: Manhattan and Euclidean distances. Obtained results outperform the state of the art, considering the different types, variability and complexity of the TCs data set, and show the effectiveness of the proposed TC segmentation method.

Rotation and scale invariant shape context registration for remote sensing images with background variations

Multitemporal remote sensing images generally suffer from background variations, which significantly disrupt traditional region feature and descriptor abstracts, especially between pre and postdisasters, making registration by local features unreliable. Because shapes hold relatively stable information, a rotation and scale invariant shape context based on multiscale edge features is proposed. A multiscale morphological operator is adapted to detect edges of shapes, and an equivalent difference of Gaussian scale space is built to detect local scale invariant feature points along the detected edges. Then, a rotation invariant shape context with improved distance discrimination serves as a feature descriptor. For a distance shape context, a self-adaptive threshold (SAT) distance division coordinate system is proposed, which improves the discriminative property of the feature descriptor in mid-long pixel distances from the central point while maintaining it in shorter ones. To achieve rotation invariance, the magnitude of Fourier transform in one-dimension is applied to calculate angle shape context. Finally, the residual error is evaluated after obtaining thin-plate spline transformation between reference and sensed images. Experimental results demonstrate the robustness, efficiency, and accuracy of this automatic algorithm.

A robust global and local mixture distance based non-rigid point set registration

We present a robust global and local mixture distance (GLMD) based non-rigid point set registration method which consists of an alternating two-step process: correspondence estimation and transformation updating. We first define two distance features for measuring global and local structural differences between two point sets, respectively. The two distances are then combined to form a GLMD based cost matrix which provides a flexible way to estimate correspondences by minimizing global or local structural differences using a linear assignment solution. To improve the correspondence estimation and enhance the interaction between the two steps, an annealing scheme is designed to gradually change the cost minimization from local to global and the thin plate spline transformation from rigid to non-rigid during registration. We test the performance of our method in contour registration, sequence images and real images, and compare with six state-of-the-art methods where our method shows the best alignments in most scenarios.

Lung registration using automatically detected landmarks.

Accurate registration of lung CT images is inevitable for numerous clinical applications. Usually, nonlinear intensity-based methods are used. Their accuracy is typically evaluated using corresponding anatomical points (landmarks; e.g. bifurcations of bronchial and vessel trees) annotated by medical experts in the images to register. As image registration can be interpreted as correspondence finding problem, these corresponding landmarks can also be used in feature-based registration techniques. Recently, approaches for automated identification of such landmark correspondences in lung CT images have been presented. In this work, a novel combination of variational nonlinear intensity-based registration with an approach for automated landmark correspondence detection in lung CT pairs is presented and evaluated. The main blocks of the proposed hybrid intensity- and feature-based registration scheme are a two-step landmark correspondence detection and the so-called CoLD (Combining Landmarks and Distance Measures) framework. The landmark correspondence identification starts with feature detection in one image followed by a blockmatching-based transfer of the features to the other image. The established correspondences are used to compute a thin-plate spline (TPS) transformation. Within CoLD, the TPS transformation is improved by minimization of an objective function consisting of a Normalized Gradient Field distance measure and a curvature regularizer; the landmark correspondences are guaranteed to be preserved by optimization on the kernel of the discretized landmark constraints. Based on ten publicly available end-inspiration/expiration CT scan pairs with anatomical landmark sets annotated by medical experts from the DIR-Lab database, it is shown that the hybrid registration approach is superior in terms of accuracy: The mean distance of expert landmarks is decreased from 8.46 mm before to 1.15 mm after registration, outperforming both the TPS transformation (1.68 mm) and a nonlinear registration without usage of automatically detected landmarks (2.44 mm). The improvement is statistically significant in eight of ten datasets in comparison to TPS and in nine of ten datasets in comparison to the intensity-based registration. Furthermore, CoLD globally estimates the breathing-induced lung volume change well and results in smooth and physiologically plausible motion fields of the lungs. We demonstrated that our novel landmark-based registration pipeline outperforms both TPS and the underlying nonlinear intensity-based registration without landmark usage. This highlights the potential of automatic landmark correspondence detection for improvement of lung CT registration accuracy.

Morphometric analysis of anatomical implant forms for minimally invasive acetabular fracture osteosynthesis

Introduction: Anatomical implants enable minimally invasive osteosynthesis (MIO) and represent ideal complements of computer-assisted surgical workflows. This 3D morphometric study analyzes anatomical implant forms (AIF) for acetabular fracture osteosynthesis (AFO).Materials and Methods: Three-dimensional pelvis models were created from clinical CT data of 99 European-Caucasian patients (50 females, 49 males). The mean age of the patients was 60.1 years (range: 20–89; SD 10.8). Definition of a referential region of interest (ROI) corresponding to an AIF for AFO was followed by automated ROI computation for each of the 198 hemipelvises. Three-dimensional statistical modeling and analysis of the resulting 198 homologous ROIs consisted of thin-plate spline transformation, generalized Procrustes fit, and principal component analysis.Results: The mean ROI length was 18.2 cm (range: 16.1–20.1 cm; SD 0.76). The first principal component (PC1) mainly modeled the ROI length, which correlated well with body height (r = 0.325; p < 0.001). PC1 comprised 47.4% of the overall ROI form variation. PC2 primarily influenced the ROI curvature in the anterior-posterior (inlet) view. Curvatures were more pronounced in female patients compared to males (p < 0.001). There was no gender-specific ROI size variation. PC1-4 contained 80.2% of the total ROI form variation. Left and right ROI forms displayed symmetry.Conclusion: This 3D morphometric study demonstrates the feasibility of anatomical implants for minimally invasive acetabular fracture osteosynthesis. Implant size/length is by far the most important variable of form variation. The necessity of gender-specific implant forms requires further investigation. The non-fractured, contralateral hemipelvis can be used for preoperative surgical planning. Ultimately, the plate design will depend on prospective implant fit tests based on the required fit as defined by the clinician.

Geometric quality enhancement of legacy graphical cadastral datasets through thin plate splines transformation

In some parts of the world, mainly developing countries, analogue graphical cadastral datasets are still being used, despite the fact that they usually contain large and differential positional distortions. This is because not only do they contain useful information, their replacement would involve lengthy legal procedures, huge financial costs and technical challenges as well. This paper presents a novel approach for the conflation of such legacy graphical cadastral datasets to more reliable and accurate topographical datasets based on the matching of linear features via a non-rigid transformation using thin plate spline transformation. The approach consists of three key steps: extraction of road network which is potentially contained in the graphical cadastral maps; matching of corresponding point and linear features between the extracted road network and a more reliable road network dataset; and computation of the non-rigid transformation between the datasets. Applying this reliable and accurate transformation quantification to the original legacy graphical cadastral dataset showed significant improvement in its geometric quality and positional accuracy.

A generalized search scheme for automatic registration of remote-sensing data

A generalized search scheme for automatic registration of level-1 data of multiresolution and multi-sensor remote-sensing data was proposed. The robustness and time efficiency of the automatic registration process is critically dependant on the search scheme for identification of control points. The proposed method consists of three levels of search ranging from coarser to finer. This process was found to be capable of registering images to sub-pixel level which are independent of rotation and scale variations, and also translation that vary by few metres to kilometres. In order to reduce the low pass effect due to multiple transformations involved in the multi-level registration process, a direct correspondence between the reference image and target image was established so that a single resampling needs to be applied. This correspondence also helps to generate products at any desired pixel size and to keep the original resolution intact. In this scheme mutual information (MI) is used as a similarity measure and a non-rigid transformation, thin plate spline (TPS), is used for achieving sub-pixel registration accuracies. MI is found to be better for identification of match points even for images that are radiometrically nonlinear. Unlike global transformation methods, use of non-rigid transformations such as TPS achieves sub-pixel accuracy in the moderate hilly regions as well as high hilly regions where relief displacements are high, provided sufficient number of control points are generated. However TPS transformation demands accurate control points. A robust method for detection of inaccurate control points was adopted and explained in the paper. The scheme was tested on a number of combinations of remote sensing data of the same resolution and different resolution datasets, namely Cartosat-1 with Liss-4 of Resourcesat-1, Landsat Thematic Mapper (TM) with IRS-1C/1D and Cartosat-1 with Enhanced Thematic Mapper (ETM). The results are presented along with accuracies achieved.

Geometric modeling and motion analysis of the epicardial surface of the heart

Pathological processes cause abnormal regional motions of the heart. Regional wall motion analyses are important to evaluate the success of therapy, especially of cell therapy, since the recovery of the heart in cell therapy proceeds slowly and results in only small changes of ventricular wall motility. The usual ultrasound imaging of heart motion is too inaccurate to be considered as an appropriate method. MRI studies are more accurate, but insufficient to reliably detect small changes in regional ventricular wall motility. We thus aim at a more accurate method of motion analysis. Our approach is based on two imaging modalities, viz. cardiac CT and biplane cineangiography. The epicardial surface represented in the CT data set at the end of the diastole is registered to the three-dimensionally reconstructed epicardial artery tree from the angiograms in end-diastolic position. The motion tracking procedures are carried out by applying thin-plate spline transformations between the epicardial artery trees belonging to consecutive frames of our cineangiographic imagery.