Landmark Annotation Research Articles

Multi-staged Feature-Attentive Network for Fashion Clothing Classification and Attribute Prediction

In the visual fashion clothing analysis, many researchers are attracted with the success of deep learning concepts. In this work, we introduce a multi-staged feature-attentive network to attain clothing category classification and attribute prediction. The proposed network in this work brings out a landmark-independent structure, whereas the existing landmark-dependent structures take up a lot of manpower for landmark annotation and also suffers from inter- and intra-individual variability. Our focus on this work is intensifying feature extraction by incorporating low-level and high-level feature fusion within fashion network. We are aiming on multi-level contextual features which utilise spatial and channel-wise information to create contextual feature supervision. Further, we enclose a semi-supervised learning approach to escalate fashion clothes analysis that utilises knowledge sharing among labelled and unlabelled data. To the best of our knowledge, this is the first attempt to investigate the semi-supervised learning in fashion clothing analysis by adopting multitask architecture which simultaneously study the clothing categories as well as its attributes. We evaluated the proposed approach on large-scale DeepFashion-C dataset while unlabelled dataset obtained from six publicly available fashion datasets. Experimental results show that the proposed architectures for supervised and semi-supervised learning entailing deep convolutional neural network outperforms many state-of-the-art techniques considerably, in fashion clothing analysis.

MetaCloth: Learning Unseen Tasks of Dense Fashion Landmark Detection From a Few Samples.

Recent advanced methods for fashion landmark detection are mainly driven by training convolutional neural networks on large-scale fashion datasets, which has a large number of annotated landmarks. However, such large-scale annotations are difficult and expensive to obtain in real-world applications, thus models that can generalize well from a small amount of labelled data are desired. We investigate this problem of few-shot fashion landmark detection, where only a few labelled samples are available for an unseen task. This work proposes a novel framework named MetaCloth via meta-learning, which is able to learn unseen tasks of dense fashion landmark detection with only a few annotated samples. Unlike previous meta-learning work that focus on solving " N -way K -shot" tasks, where each task predicts N number of classes by training with K annotated samples for each class ( N is fixed for all seen and unseen tasks), a task in MetaCloth detects N different landmarks for different clothing categories using K samples, where N varies across tasks, because different clothing categories usually have various number of landmarks. Therefore, numbers of parameters are various for different seen and unseen tasks in MetaCloth. MetaCloth is carefully designed to dynamically generate different numbers of parameters for different tasks, and learn a generalizable feature extraction network from a few annotated samples with a set of good initialization parameters. Extensive experiments show that MetaCloth outperforms its counterparts by a large margin.

Use of Advanced Artificial Intelligence in Forensic Medicine, Forensic Anthropology and Clinical Anatomy.

Three-dimensional convolutional neural networks (3D CNN) of artificial intelligence (AI) are potent in image processing and recognition using deep learning to perform generative and descriptive tasks. Compared to its predecessor, the advantage of CNN is that it automatically detects the important features without any human supervision. 3D CNN is used to extract features in three dimensions where input is a 3D volume or a sequence of 2D pictures, e.g., slices in a cone-beam computer tomography scan (CBCT). The main aim was to bridge interdisciplinary cooperation between forensic medical experts and deep learning engineers, emphasizing activating clinical forensic experts in the field with possibly basic knowledge of advanced artificial intelligence techniques with interest in its implementation in their efforts to advance forensic research further. This paper introduces a novel workflow of 3D CNN analysis of full-head CBCT scans. Authors explore the current and design customized 3D CNN application methods for particular forensic research in five perspectives: (1) sex determination, (2) biological age estimation, (3) 3D cephalometric landmark annotation, (4) growth vectors prediction, (5) facial soft-tissue estimation from the skull and vice versa. In conclusion, 3D CNN application can be a watershed moment in forensic medicine, leading to unprecedented improvement of forensic analysis workflows based on 3D neural networks.

Multilingual speech annotation of landmarks and other acoustic cues to distinctive features

A speech annotation system developed for English that identifies landmarks and other acoustic cues to distinctive features (Stevens, 2002; Huilgol et al., 2019) has been extended to Spanish and Korean. This process includes retrieving the (allo)phone sequence for the words of a target utterance from a standard lexicon, converting this sequence into an underlying phoneme sequence, and then generating citation-form landmark and other acoustic cue sequences from the phoneme sequence and its features. This standard set is presented to labelers who examine the waveform of the target utterance and mark which of the predicted cues have been realized, or deleted, and where unpredicted cues have been inserted. For each language, we draw from a superset of acoustic cues that function in all three languages, but each language also makes use of an additional set of acoustic cues specific to that language.

Modeling Annotation Uncertainty with Gaussian Heatmaps in Landmark Localization

In landmark localization, due to ambiguities in defining their exact position, landmark annotations may suffer from large observer variabilities, which result in uncertain annotations. To model the annotation ambiguities of the training dataset, we propose to learn anisotropic Gaussian parameters modeling the shape of the target heatmap during optimization. Furthermore, our method models the prediction uncertainty of individual samples by fitting anisotropic Gaussian functions to the predicted heatmaps during inference. Besides state-of-the-art results, our experiments on datasets of hand radiographs and lateral cephalograms also show that Gaussian functions are correlated with both localization accuracy and observer variability. As a final experiment, we show the importance of integrating the uncertainty into decision making by measuring the influence of the predicted location uncertainty on the classification of anatomical abnormalities in lateral cephalograms.

Automatic landmark annotation in 3D surface scans of skulls: Methodological proposal and reliability study

Background and Objectives: Craniometric landmarks are essential in many biomedical applications, such as morphometric analysis or forensic identification. The process of locating landmarks is usually a manual and slow task, highly influenced by fatigue, skills and the experience of the practitioner. Localization errors are propagated and magnified in subsequent steps, which can result in incorrect measurements or assumptions. Thereby, standardization, reliability and reproducibility lay the foundations for the necessary accuracy in subsequent measurements or anatomical analysis. In this paper, we present an automatic method to annotate 3D surface skull models taking into account anatomical and geometrical features.Methods: The proposed method follows a hybrid structure where a deformable template is used to initialize the landmark positions. Then, a refinement stage is applied using prior anatomical knowledge to ensure a correct placement. Our proposal is validated over thirty 3D skull scans of male Caucasians, acquired by hand-held surface scanning, and a set of 58 craniometric landmarks. A statistical analysis was carried out to analyze the inter- and intra-observer variability of manual annotations and the automatic results, along with a visual assessment of the final results.Results: Inter-observer errors show significant differences, which are reflected in the expert consensus used as reference. The average localization error was 2.19±1.5 mm when comparing the automatic landmarks to the reference location. The subsequent visual analysis confirmed the reliability of the refinement method for most landmarks.Conclusions: Repeated manual annotations show a high variability depending on both skills and expertise of the observer, and landmarks’ location and characteristics. In contrast, the automatic method provides an accurate, robust and reproducible alternative to the tedious and error-prone task of manual landmarking.

Automated landmarking for palatal shape analysis using geometric deep learning.

To develop and evaluate a geometric deep-learning network to automatically place seven palatal landmarks on digitized maxillary dental casts. The sample comprised individuals with permanent dentition of various ethnicities. The network was trained from manual landmark annotations on 732 dental casts and evaluated on 104 dental casts. A geometric deep-learning network was developed to hierarchically learn features from point-clouds representing the 3D surface of each cast. These features predict the locations of seven palatal landmarks. Repeat-measurement reliability was <0.3mm for all landmarks on all casts. Accuracy is promising. The proportion of test subjects with errors less than 2mm was between 0.93 and 0.68, depending on the landmark. Unusually shaped and large palates generate the highest errors. There was no evidence for a difference in mean palatal shape estimated from manual compared to the automatic landmarking. The automatic landmarking reduces sample variation around the mean and reduces measurements of palatal size. The automatic landmarking method shows excellent repeatability and promising accuracy, which can streamline patient assessment and research studies. However, landmark indications should be subject to visual quality control.

Machine Learning-Guided Prediction of Central Anterior Chamber Depth Using Slit Lamp Images from a Portable Smartphone Device.

There is currently no objective portable screening modality for narrow angles in the community. In this prospective, single-centre image validation study, we used machine learning on slit lamp images taken with a portable smartphone device (MIDAS) to predict the central anterior chamber depth (ACD) of phakic patients with undilated pupils. Patients 60 years or older with no history of laser or intraocular surgery were recruited. Slit lamp images were taken with MIDAS, followed by anterior segment optical coherence tomography (ASOCT; Casia SS-1000, Tomey, Nagoya, Japan). After manual annotation of the anatomical landmarks of the slit lamp photos, machine learning was applied after image processing and feature extraction to predict the ACD. These values were then compared with those acquired from the ASOCT. Sixty-six eyes (right = 39, 59.1%) were included for analysis. The predicted ACD values formed a strong positive correlation with the measured ACD values from ASOCT (R2 = 0.91 for training data and R2 = 0.73 for test data). This study suggests the possibility of estimating central ACD using slit lamp images taken from portable devices.

Deep Learning Based Airway Segmentation Using Key Point Prediction

The purpose of this study was to investigate the accuracy of the airway volume measurement by a Regression Neural Network-based deep-learning model. A set of manually outlined airway data was set to build the algorithm for fully automatic segmentation of a deep learning process. Manual landmarks of the airway were determined by one examiner using a mid-sagittal plane of cone-beam computed tomography (CBCT) images of 315 patients. Clinical dataset-based training with data augmentation was conducted. Based on the annotated landmarks, the airway passage was measured and segmented. The accuracy of our model was confirmed by measuring the following between the examiner and the program: (1) a difference in volume of nasopharynx, oropharynx, and hypopharynx, and (2) the Euclidean distance. For the agreement analysis, 61 samples were extracted and compared. The correlation test showed a range of good to excellent reliability. A difference between volumes were analyzed using regression analysis. The slope of the two measurements was close to 1 and showed a linear regression correlation (r2 = 0.975, slope = 1.02, p &lt; 0.001). These results indicate that fully automatic segmentation of the airway is possible by training via deep learning of artificial intelligence. Additionally, a high correlation between manual data and deep learning data was estimated.

2D Wasserstein loss for robust facial landmark detection

The recent performance of facial landmark detection has been significantly improved by using deep Convolutional Neural Networks (CNNs), especially the Heatmap Regression Models (HRMs). Although their performance on common benchmark datasets has reached a high level, the robustness of these models still remains a challenging problem in the practical use under noisy conditions of realistic environments. Contrary to most existing work focusing on the design of new models, we argue that improving the robustness requires rethinking many other aspects, including the use of datasets, the format of landmark annotation, the evaluation metric as well as the training and detection algorithm itself. In this paper, we propose a novel method for robust facial landmark detection, using a loss function based on the 2D Wasserstein distance combined with a new landmark coordinate sampling relying on the barycenter of the individual probability distributions. Our method can be plugged-and-play on most state-of-the-art HRMs with neither additional complexity nor structural modifications of the models. Further, with the large performance increase, we found that current evaluation metrics can no longer fully reflect the robustness of these models. Therefore, we propose several improvements to the standard evaluation protocol. Extensive experimental results on both traditional evaluation metrics and our evaluation metrics demonstrate that our approach significantly improves the robustness of state-of-the-art facial landmark detection models.

Implement of the transformation for markerless facial motion capture in 3D animation

Motion capture is a motion assisting technology that can resemble the motion of objects being captured and is now widely used for film development, especially in visual effects or animation. From that, the author has the idea, namely to use a Blender application to implement a markerless facial motion capture on motion of object 3D modeling based on webcam because it didn’t have a menu or feature for markerless motion capture. Markerless is meant to process the assignment of annotation points to objects with a computer system so that this method is very simple and saves time in its operation, especially in facial motion capture. In this research, the face detection method uses haar cascade and to provide facial landmark annotation using detector landmark lbf model. We are using an example of 3D modeling a human face that has a bone on the neck, head, mouth, eyes, and eyebrows that will follow the movement of the human face. The combination of face landmark detection and rigging in 3D modeling a human face uses the transformation geometry method to produce real-time motion. From this research study, 3D animated face objects can follow motion on real face targets but there is still noise.

IE-Map: a novel in-vivo atlas and template of the human inner ear

Brain atlases and templates are core tools in scientific research with increasing importance also in clinical applications. Advances in neuroimaging now allowed us to expand the atlas domain to the vestibular and auditory organ, the inner ear. In this study, we present IE-Map, an in-vivo template and atlas of the human labyrinth derived from multi-modal high-resolution magnetic resonance imaging (MRI) data, in a fully non-invasive manner without any contrast agent or radiation. We reconstructed a common template from 126 inner ears (63 normal subjects) and annotated it with 94 established landmarks and semi-automatic segmentations of all relevant macroscopic vestibular and auditory substructures. We validated the atlas by comparing MRI templates to a novel CT/micro-CT atlas, which we reconstructed from 21 publicly available post-mortem images of the bony labyrinth. Templates in MRI and micro-CT have a high overlap, and several key anatomical measures of the bony labyrinth in IE-Map are in line with micro-CT literature of the inner ear. A quantitative substructural analysis based on the new template, revealed a correlation of labyrinth parameters with total intracranial volume. No effects of gender or laterality were found. We provide the validated templates, atlas segmentations, surface meshes and landmark annotations as open-access material, to provide neuroscience researchers and clinicians in neurology, neurosurgery, and otorhinolaryngology with a widely applicable tool for computational neuro-otology.

Feasibility of Cochlea High-frequency Ultrasound and Microcomputed Tomography Registration for Cochlear Computer-assisted Surgery: A Testbed.

There remains no standard imaging method that allows computer-assisted surgery of the cochlea in real time. However, recent evidence suggests that high-frequency ultrasound (HFUS) could permit real-time visualization of cochlear architecture. Registration with an imaging modality that suffers neither attenuation nor conical deformation could reveal useful anatomical landmarks to surgeons. Our study aimed to address the feasibility of an automated three-dimensional (3D) HFUS/microCT registration, and to evaluate the identification of cochlear structures using 2D/3D HFUS and microCT. MicroCT, and 2D/3D 40 MHz US in B-mode were performed on ex vivo guinea pig cochlea. An automatic rigid registration algorithm was applied to segmented 3D images. This automatic registration was then compared to a reference method using manual annotated landmarks placed by two senior otologists. Inter- and intrarater reliabilities were evaluated using intraclass correlation coefficient (ICC) and the mean registration error was calculated. 3D HFUS/microCT automatic registration was successful. Excellent levels of concordance were achieved with regards intra-rater reliability for both raters with micro-CT and US images (ICC ranging from 0.98 to 1, p < 0.001) and with regards inter-rater reliability (ICC ranging from 0.99 to 1, p < 0.001). The mean HFUS/microCT automated RE for both observers was 0.17 ± 0.03 mm [0.10-0.25]. Identification of the basilar membrane, modiolus, scala tympani, and scala vestibuli was possible with 2D/3D HFUS and micro-CT. HFUS/microCT image registration is feasible. 2D/3D HFUS and microCT allow the visualization of cochlear structures. Many potential clinical applications are conceivable.

Facial imaging and landmark detection technique for objective assessment of unilateral peripheral facial paralysis

ABSTRACT In this paper, we propose a hypothesis that the facial landmark detection methods constructed by a private UPFP facial dataset can perform better than the model on a healthy facial dataset in the task of UPFP facial landmark detection. For proving this hypothesis, a customized UPFP facial dataset with 68 facial landmark annotations was built. A state-of-the-art facial landmark detection method was employed on the three evaluation datasets to exploit and prove the hypothesis. The mean error of validation dataset is 3.15, 56% lower than 7.42 that of the healthy dataset, which proves the hypothesis is true.

Semi-Supervised Landmark-Guided Restoration of Atmospheric Turbulent Images

Image degradation due to atmospheric turbulence (AT), which is common while capturing images at long ranges, adversely affects the performance of tasks such as face alignment and face recognition. To the best of our knowledge, there does not exist any dataset consisting of turbulence-degraded face images along with their annotated landmarks and ground-truth clean images, making supervised training challenging. In this paper, we present a semisupervised method for jointly extracting facial landmarks and restoring the degraded images by exploiting the semantic information from the landmarks. The proposed approach learns to generate AT images by combining the content from a clean image and turbulence information from AT images in an unpaired manner. Next, we use heatmaps from the landmark localization network as a prior to the image restoration module. Subsequently, we impose heatmap consistency loss and heatmap confidence loss to regularize the restored images. Extensive experiments demonstrate the effectiveness of the proposed network, which achieves an NME of 2.797 on the task of landmark localization for strong turbulent images and yields improved restoration results compared to state-of-the-art methods.

CASIA-Face-Africa: A Large-Scale African Face Image Database

Face recognition is a popular and well-studied area with wide applications in our society. However, racial bias had been proven to be inherent in most State Of The Art (SOTA) face recognition systems. Many investigative studies on face recognition algorithms have reported higher false positive rates of African subjects cohorts than the other cohorts. Lack of large-scale African face image databases in public domain is one of the main restrictions in studying the racial bias problem of face recognition. To this end, we collect a face image database namely CASIA-Face-Africa which contains 38,546 images of 1,183 African subjects. Multi-spectral cameras are utilized to capture the face images under various illumination settings. Demographic attributes and facial expressions of the subjects are also carefully recorded. For landmark detection, each face image in the database is manually labeled with 68 facial keypoints. A group of evaluation protocols are constructed according to different applications, tasks, partitions and scenarios. The performances of SOTA face recognition algorithms without re-training are reported as baselines. The proposed database along with its face landmark annotations, evaluation protocols and preliminary results form a good benchmark to study the essential aspects of face biometrics for African subjects, especially face image preprocessing, face feature analysis and matching, facial expression recognition, sex/age estimation, ethnic classification, face image generation, etc. The database can be downloaded from our website.

Landmark annotation and mandibular lateral deviation analysis of posteroanterior cephalograms using a convolutional neural network

Background/purposeFacial asymmetry is relatively common in the general population. Here, we propose a fully automated annotation system that supports analysis of mandibular deviation and detection of facial asymmetry in posteroanterior (PA) cephalograms by means of a deep learning-based convolutional neural network (CNN) algorithm. Materials and methodsIn this retrospective study, 400 PA cephalograms were collected from the medical records of patients aged 4 years 2 months–80 years 3 months (mean age, 17 years 10 months; 255 female patients and 145 male patients). A deep CNN with two optimizers and a random forest algorithm were trained using 320 PA cephalograms; in these images, four PA landmarks were independently identified and manually annotated by two orthodontists. ResultsThe CNN algorithms had a high coefficient of determination (R2), compared with the random forest algorithm (CNN-stochastic gradient descent, R2 = 0.715; CNN-Adam, R2 = 0.700; random forest, R2 = 0.486). Analysis of the best and worst performances of the algorithms for each landmark demonstrated that the right latero-orbital landmark was most difficult to detect accurately by using the CNN. Based on the annotated landmarks, reference lines were defined using an algorithm coded in Python. The CNN and random forest algorithms exhibited similar accuracy for the distance between the menton and vertical reference line. ConclusionOur findings imply that the proposed deep CNN algorithm for detection of facial asymmetry may enable prompt assessment and reduce the effort involved in orthodontic diagnosis.

A New Dataset for Facial Motion Analysis in Individuals With Neurological Disorders.

We present the first public dataset with videos of oro-facial gestures performed by individuals with oro-facial impairment due to neurological disorders, such as amyotrophic lateral sclerosis (ALS) and stroke. Perceptual clinical scores from trained clinicians are provided as metadata. Manual annotation of facial landmarks is also provided for a subset of over 3300 frames. Through extensive experiments with multiple facial landmark detection algorithms, including state-of-the-art convolutional neural network (CNN) models, we demonstrated the presence of bias in the landmark localization accuracy of pre-trained face alignment approaches in our participant groups. The pre-trained models produced higher errors in the two clinical groups compared to age-matched healthy control subjects. We also investigated how this bias changes when the existing models are fine-tuned using data from the target population. The release of this dataset aims to propel the development of face alignment algorithms robust to the presence of oro-facial impairment, support the automatic analysis and recognition of oro-facial gestures, enhance the automatic identification of neurological diseases, as well as the estimation of disease severity from videos and images.

Learning-based local-to-global landmark annotation for automatic 3D cephalometry

The annotation of three-dimensional (3D) cephalometric landmarks in 3D computerized tomography (CT) has become an essential part of cephalometric analysis, which is used for diagnosis, surgical planning, and treatment evaluation. The automation of 3D landmarking with high-precision remains challenging due to the limited availability of training data and the high computational burden. This paper addresses these challenges by proposing a hierarchical deep-learning method consisting of four stages: 1) a basic landmark annotator for 3D skull pose normalization, 2) a deep-learning-based coarse-to-fine landmark annotator on the midsagittal plane, 3) a low-dimensional representation of the total number of landmarks using variational autoencoder (VAE), and 4) a local-to-global landmark annotator. The implementation of the VAE allows two-dimensional-image-based 3D morphological feature learning and similarity/dissimilarity representation learning of the concatenated vectors of cephalometric landmarks. The proposed method achieves an average 3D point-to-point error of 3.63 mm for 93 cephalometric landmarks using a small number of training CT datasets. Notably, the VAE captures variations of craniofacial structural characteristics.

ML‐morph: A fast, accurate and general approach for automated detection and landmarking of biological structures in images

Abstract Morphometrics has become an indispensable component of the statistical analysis of size and shape variation in biological structures. Morphometric data have traditionally been gathered through low‐throughput manual landmark annotation, which represents a significant bottleneck for morphometric‐based phenomics. Here we propose a machine‐learning‐based high‐throughput pipeline to collect high‐dimensional morphometric data in two‐dimensional images of semi‐rigid biological structures. The proposed framework has four main strengths. First, it allows for dense phenotyping with minimal impact on specimens. Second, it presents landmarking accuracy comparable to manual annotators, when applied to standardized datasets. Third, it performs data collection at speeds several orders of magnitude higher than manual annotators. And finally, it is of general applicability (i.e. not tied to a specific study system). State‐of‐the‐art validation procedures show that the method achieves low error levels when applied to three morphometric datasets of increasing complexity, with error varying from 0.57% to 2.2% of the structure's length in the automated placement of landmarks. As a benchmark for the speed of the entire automated landmarking pipeline, our framework places 23 landmarks on 13,686 objects (zooids) detected in 1,684 pictures of fossil bryozoans in 3.12 min using a personal computer. The proposed machine‐learning‐based phenotyping pipeline can greatly increase the scale, reproducibility and speed of data collection within biological research. To aid the use of the framework, we have developed a file conversion algorithm that can be used to leverage current morphometric datasets for automation, allowing the entire procedure, from model training all the way to prediction, to be performed in a matter of hours.