Landmark Detection Research Articles

A smartphone application for personalized facial aesthetic monitoring.

Methods available at home for capturing facial images to track changes in skin quality and evaluate skincare treatments are limited. In this study, we developed a smartphone camera application (app) for personalized facial aesthetic monitoring. A face alignment indicators (FAIN) system utilizing facial landmark detection, an artificial intelligence technique, to estimate key facial parts, was implemented into the app to maintain a consistent facial appearance during image capture. The FAIN system is composed of a fixed target indicator and an alignment indicator that dynamically changes its shape according to the user's face position, size, and orientation. Users align their faces to match the alignment indicator with the fixed target indicator, and the image is automatically captured when alignment is achieved. We investigated the app's effectiveness in ensuring a consistent facial appearance by analyzing both geometric and colorimetric data. Geometric information from captured faces and colorimetric data from stickers applied to the faces were utilized. The coefficients of variation (CVs) for the L*, a*, and b* values of the stickers were higher compared to those measured by a colorimeter, with CVs of 14.9 times, 8.14 times, and 4.41 times for L*, a*, and b*, respectively. To assess the feasibility of the app for facial aesthetic monitoring, we tracked changes in pseudo-skin color on the cheek of a participant using skin-colored stickers. As a result, we observed the smallest color difference ∆Eab of 1.901, which can be considered as the experimentally validated detection limit using images acquired by the app. While the current monitoring method is a relative quantification approach, it contributes to evidence-based evaluations of skincare treatments.

Multi-Sensor-Assisted Low-Cost Indoor Non-Visual Semantic Map Construction and Localization for Modern Vehicles.

With the transformation and development of the automotive industry, low-cost and seamless indoor and outdoor positioning has become a research hotspot for modern vehicles equipped with in-vehicle infotainment systems, Internet of Vehicles, or other intelligent systems (such as Telematics Box, Autopilot, etc.). This paper analyzes modern vehicles in different configurations and proposes a low-cost, versatile indoor non-visual semantic mapping and localization solution based on low-cost sensors. Firstly, the sliding window-based semantic landmark detection method is designed to identify non-visual semantic landmarks (e.g., entrance/exit, ramp entrance/exit, road node). Then, we construct an indoor non-visual semantic map that includes the vehicle trajectory waypoints, non-visual semantic landmarks, and Wi-Fi fingerprints of RSS features. Furthermore, to estimate the position of modern vehicles in the constructed semantic maps, we proposed a graph-optimized localization method based on landmark matching that exploits the correlation between non-visual semantic landmarks. Finally, field experiments are conducted in two shopping mall scenes with different underground parking layouts to verify the proposed non-visual semantic mapping and localization method. The results show that the proposed method achieves a high accuracy of 98.1% in non-visual semantic landmark detection and a low localization error of 1.31 m.

Can artificial intelligence-driven cephalometric analysis replace manual tracing? A systematic review and meta-analysis.

This systematic review and meta-analysis aimed to investigate the accuracy and efficiency of artificial intelligence (AI)-driven automated landmark detection for cephalometric analysis on two-dimensional (2D) lateral cephalograms and three-dimensional (3D) cone-beam computed tomographic (CBCT) images. An electronic search was conducted in the following databases: PubMed, Web of Science, Embase, and grey literature with search timeline extending up to January 2024. Studies that employed AI for 2D or 3D cephalometric landmark detection were included. The selection of studies, data extraction, and quality assessment of the included studies were performed independently by two reviewers. The risk of bias was assessed using the Quality Assessment of Diagnostic Accuracy Studies-2 tool. A meta-analysis was conducted to evaluate the accuracy of the 2D landmarks identification based on both mean radial error and standard error. Following the removal of duplicates, title and abstract screening, and full-text reading, 34 publications were selected. Amongst these, 27 studies evaluated the accuracy of AI-driven automated landmarking on 2D lateral cephalograms, while 7 studies involved 3D-CBCT images. A meta-analysis, based on the success detection rate of landmark placement on 2D images, revealed that the error was below the clinically acceptable threshold of 2mm (1.39mm; 95% confidence interval: 0.85-1.92mm). For 3D images, meta-analysis could not be conducted due to significant heterogeneity amongst the study designs. However, qualitative synthesis indicated that the mean error of landmark detection on 3D images ranged from 1.0 to 5.8mm. Both automated 2D and 3D landmarking proved to be time-efficient, taking less than 1min. Most studies exhibited a high risk of bias in data selection (n = 27) and reference standard (n = 29). The performance of AI-driven cephalometric landmark detection on both 2D cephalograms and 3D-CBCT images showed potential in terms of accuracy and time efficiency. However, the generalizability and robustness of these AI systems could benefit from further improvement. PROSPERO: CRD42022328800.

Use of Yolo Detection for 3D Pose Tracking of Cardiac Catheters Using Bi-Plane Fluoroscopy

The increasing rate of minimally invasive procedures and the growing prevalence of cardiovascular disease have led to a demand for higher-quality guidance systems for catheter tracking. Traditional methods for catheter tracking, such as detection based on single points and applying masking techniques, have been limited in their ability to provide accurate pose information. In this paper, we propose a novel deep learning-based method for catheter tracking and pose detection. Our method uses a Yolov5 bounding box neural network with postprocessing to perform landmark detection in four regions of the catheter: the tip, radio-opaque marker, bend, and entry point. This allows us to track the catheter’s position and orientation in real time, without the need for additional masking or segmentation techniques. We evaluated our method on a dataset of fluoroscopic images from two distinct datasets and achieved state-of-the-art results in terms of accuracy and robustness. Our model was able to detect all four landmark features (tip, marker, bend, and entry) used to generate a pose for a catheter with 0.285 ± 0.143 mm, 0.261 ± 0.138 mm, 0.424 ± 0.361 mm, and 0.235 ± 0.085 mm accuracy. We believe that our method has the potential to significantly improve the accuracy and efficiency of catheter tracking in medical procedures that utilize bi-plane fluoroscopy guidance.

Key-Region-Based UAV Visual Navigation

Abstract. Visual navigation has recently seen significant developments with the rise in autonomous navigation. Keypoint-based mapping and localization has served as a reliable localization method for many applications, but the push to run more applications on less expensive hardware becomes extremely limiting. In this paper, we present a novel approach for visual geolocalization and navigation that improves landmark detection reliability while reducing reference map complexity. Similar to prior techniques, we use the process of point based matching schemes to solve for the image-to-map transform. The critical difference is that we use object detection to identify key-regions instead of keypoints. During an initial flight key-regions are mapped into an identity dictionary with their geolocations and few-shot learning encoded descriptors. Then on subsequent flights, key-regions are detected and matched using the identity dictionary for re-identification. Using the identified vehicles as key-regions, the results show that the proposed key-region based localization produces GPS like localization while maintaining a higher resilience to image noise compared to keypoint-based techniques.

Enhancing nasal endoscopy: Classification, detection, and segmentation of anatomic landmarks using a convolutional neural network.

A convolutional neural network (CNN)-based model can accurately localize and segment turbinates in images obtained during nasal endoscopy (NE). This model represents a starting point for algorithms that comprehensively interpret NE findings.

Infinite 3D Landmarks: Improving Continuous 2D Facial Landmark Detection

AbstractIn this paper, we examine three important issues in the practical use of state‐of‐the‐art facial landmark detectors and show how a combination of specific architectural modifications can directly improve their accuracy and temporal stability. First, many facial landmark detectors require a face normalization step as a pre‐process, often accomplished by a separately trained neural network that crops and resizes the face in the input image. There is no guarantee that this pre‐trained network performs optimal face normalization for the task of landmark detection. Thus, we instead analyse the use of a spatial transformer network that is trained alongside the landmark detector in an unsupervised manner, jointly learning an optimal face normalization and landmark detection by a single neural network. Second, we show that modifying the output head of the landmark predictor to infer landmarks in a canonical 3D space rather than directly in 2D can further improve accuracy. To convert the predicted 3D landmarks into screen‐space, we additionally predict the camera intrinsics and head pose from the input image. As a side benefit, this allows to predict the 3D face shape from a given image only using 2D landmarks as supervision, which is useful in determining landmark visibility among other things. Third, when training a landmark detector on multiple datasets at the same time, annotation inconsistencies across datasets forces the network to produce a sub‐optimal average. We propose to add a semantic correction network to address this issue. This additional lightweight neural network is trained alongside the landmark detector, without requiring any additional supervision. While the insights of this paper can be applied to most common landmark detectors, we specifically target a recently proposed continuous 2D landmark detector to demonstrate how each of our additions leads to meaningful improvements over the state‐of‐the‐art on standard benchmarks.

Accuracy of a deep learning-based algorithm for the detection of thoracic aortic calcifications in chest computed tomography and cardiovascular surgery planning.

To assess the accuracy of a deep learning-based algorithm for fully automated detection of thoracic aortic calcifications in chest computed tomography (CT) with a focus on the aortic clamping zone. We retrospectively included 100 chest CT scans from 91 patients who were examined on second- or third-generation dual-source scanners. Subsamples comprised 47 scans with an ECG-gated aortic angiography and 53 unenhanced scans. A deep learning model performed aortic landmark detection and aorta segmentation to derive eight vessel segments. Associated calcifications were detected and their volumes measured using a mean-based density thresholding. Algorithm parameters (calcium cluster size threshold, aortic mask dilatation) were varied to determine optimal performance for the upper ascending aorta that encompasses the aortic clamping zone. A binary visual rating served as a reference. Standard estimates of diagnostic accuracy and inter-rater agreement using Cohen's Kappa were calculated. Thoracic aortic calcifications were observed in 74% of patients with a prevalence of 27% to 70% by aorta segment. Using different parameter combinations, the algorithm provided binary ratings for all scans and segments. The best-performing parameter combination for the presence of calcifications in the aortic clamping zone yielded a sensitivity of 93% and a specificity of 82% with an area under the receiver operating characteristic curve of 0.874. Using these parameters, the inter-rater agreement ranged from κ 0.66 to 0.92 per segment. Fully automated segmental detection of thoracic aortic calcifications in chest CT performs with high accuracy. This includes the critical preoperative assessment of the aortic clamping zone.

A Bayesian network for simultaneous keyframe and landmark detection in ultrasonic cine

Accurate landmark detection in medical imaging is essential for quantifying various anatomical structures and assisting in diagnosis and treatment planning. In ultrasound cine, landmark detection is often associated with identifying keyframes, which represent the occurrence of specific events, such as measuring target dimensions at specific temporal phases. Existing methods predominantly treat landmark and keyframe detection as separate tasks without harnessing their underlying correlations. Additionally, owing to the intrinsic characteristics of ultrasound imaging, both tasks are constrained by inter-observer variability, leading to potentially higher levels of uncertainty. In this paper, we propose a Bayesian network to achieve simultaneous keyframe and landmark detection in ultrasonic cine, especially under highly sparse training data conditions. We follow a coarse-to-fine landmark detection architecture and propose an adaptive Bayesian hypergraph for coordinate refinement on the results of heatmap-based regression. In addition, we propose Order Loss for training bi-directional Gated Recurrent Unit to identify keyframes based on the relative likelihoods within the sequence. Furthermore, to exploit the underlying correlation between the two tasks, we use a shared encoder to extract features for both tasks and enhance the detection accuracy through the interaction of temporal and motion information. Experiments on two in-house datasets (multi-view transesophageal and transthoracic echocardiography) and one public dataset (transthoracic echocardiography) demonstrate that our method outperforms state-of-the-art approaches. The mean absolute errors for dimension measurements of the left atrial appendage, aortic annulus, and left ventricle are 2.40 mm, 0.83 mm, and 1.63 mm, respectively. The source code is available at github.com/warmestwind/ABHG.

A Novel Deep Transfer Learning-Based Approach for Face Pose Estimation

Abstract An efficient face recognition system is essential for security and authentication-based applications. However, real-time face recognition systems have a few significant concerns, including face pose orientations. In the last decade, numerous solutions have been introduced to estimate distinct face pose orientations. Nevertheless, these solutions must be adequately addressed for the three main face pose orientations: Yaw, Pitch, and Roll. This paper proposed a novel deep transfer learning-based multitasking approach for solving three integrated tasks, i.e., face detection, landmarks detection, and face pose estimation. The face pose variation vulnerability has been intensely investigated here underlying three modules: image preprocessing, feature extraction module through deep transfer learning, and regression module for estimating the face poses. The experiments are performed on the well-known benchmark dataset Annotated Faces in the Wild (AFW). We evaluate the outcomes of the experiments to reveal that our proposed approach is superior to other recently available solutions.

An Automatic Approach of Video-Based Landmark Detection and Movement Analysis for Assessing Symptoms of Bradykinesia in Parkinson’s Disease

An Automatic Approach of Video-Based Landmark Detection and Movement Analysis for Assessing Symptoms of Bradykinesia in Parkinson’s Disease

Modified DSFD and TCDCN Based Facial Landmark Detection for Gender and Age Classification

Modified DSFD and TCDCN Based Facial Landmark Detection for Gender and Age Classification

Alertness Assured: Sleep Detection and Video Control Technology

Abstract: The system utilizes advanced computer vision algorithms to analyze real-time video streams, employing techniques such as facial landmark detection and eye movement tracking. By continuously monitoring these visual cues, the system can accurately detect signs of drowsiness, such as drooping eyelids or prolonged eye closures, prompting timely interventions to prevent accidents or errors caused by fatigue. In addition to detecting drowsiness, the system also evaluates the user's attention levels by analysing head orientation and facial expressions. By tracking head movements and assessing changes in facial expressions indicative of engagement or distraction, the system provides valuable insights into the user's cognitive state. This attention assessment component enables the system to adapt its feedback and intervention strategies based on the user's level of alertness and focus. To enhance user awareness and responsiveness, the system employs graphical visualization techniques to display real-time feedback on drowsiness and attention status. Visual indicators, such as color-coded alerts or dynamic graphs depicting attention trends, provide users with intuitive insights into their cognitive performance, empowering them to make informed decisions about their work habits or driving patterns. Moreover, the system utilizes auditory alerts to supplement visual feedback, ensuring that users receive timely notifications even in noisy or visually demanding environments. Whether it's a gentle reminder to take a short break or a more urgent warning about escalating drowsiness levels, auditory cues serve as an effective means of alerting users to potential risks and encouraging proactive intervention

Facial Action Unit detection based on multi-task learning strategy for unlabeled facial images in the wild

Facial Action Unit (AU) detection often relies on highly-cost accurate labeling or inaccurate pseudo labeling techniques in recent years. How to introduce large amounts of unlabeled facial images in the wild into supervised AU detection frameworks has become a challenging problem. Additionally, nearly every type of AUs has the problem of unbalanced positive and negative samples. Inspired by other multi-task learning frameworks, we first propose a multi-task learning strategy boosting AU detection in the wild through jointing facial landmark detection and AU domain separation and reconstruction. Our introduced dual domains facial landmark detection framework can solve the lack of accurate facial landmark coordinates during the AU domain separation and reconstruction training process, while the parameters of homostructural facial extraction modules from these two similar facial tasks are shared. Moreover, we propose a pixel-level feature alignment scheme to maintain the consistency of features obtained from two separation and reconstruction processes. Furthermore, a weighted asymmetric loss is proposed to change the contribution of positive and negative samples of each type of AUs to model parameters updating. Experimental results on three widely used benchmarks demonstrate our superiority to most state-of-the-art methods for AU detection.

Multitask Learning Strategy with Pseudo-Labeling: Face Recognition, Facial Landmark Detection, and Head Pose Estimation

Most facial analysis methods perform well in standardized testing but not in real-world testing. The main reason is that training models cannot easily learn various human features and background noise, especially for facial landmark detection and head pose estimation tasks with limited and noisy training datasets. To alleviate the gap between standardized and real-world testing, we propose a pseudo-labeling technique using a face recognition dataset consisting of various people and background noise. The use of our pseudo-labeled training dataset can help to overcome the lack of diversity among the people in the dataset. Our integrated framework is constructed using complementary multitask learning methods to extract robust features for each task. Furthermore, introducing pseudo-labeling and multitask learning improves the face recognition performance by enabling the learning of pose-invariant features. Our method achieves state-of-the-art (SOTA) or near-SOTA performance on the AFLW2000-3D and BIWI datasets for facial landmark detection and head pose estimation, with competitive face verification performance on the IJB-C test dataset for face recognition. We demonstrate this through a novel testing methodology that categorizes cases as soft, medium, and hard based on the pose values of IJB-C. The proposed method achieves stable performance even when the dataset lacks diverse face identifications.

Deep Learning for Predicting Progression of Patellofemoral Osteoarthritis Based on Lateral Knee Radiographs, Demographic Data, and Symptomatic Assessments.

In this study, we propose a novel framework that utilizes deep learning and attention mechanisms to predict the radiographic progression of patellofemoral osteoarthritis (PFOA) over a period of 7 years. This study included subjects (1,832 subjects, 3,276 knees) from the baseline of the Multicenter Osteoarthritis Study (MOST). Patellofemoral joint regions of interest were identified using an automated landmark detection tool (BoneFinder) on lateral knee X-rays. An end-to-end deep learning method was developed for predicting PFOA progression based on imaging data in a five-fold cross-validation setting. To evaluate the performance of the models, a set of baselines based on known risk factors were developed and analyzed using gradient boosting machine (GBM). Risk factors included age, sex, body mass index, and Western Ontario and McMaster Universities Arthritis Index score, and the radiographic osteoarthritis stage of the tibiofemoral joint (Kellgren and Lawrence [KL] score). Finally, to increase predictive power, we trained an ensemble model using both imaging and clinical data. Among the individual models, the performance of our deep convolutional neural network attention model achieved the best performance with an area under the receiver operating characteristic curve (AUC) of 0.856 and average precision (AP) of 0.431, slightly outperforming the deep learning approach without attention (AUC = 0.832, AP = 0.4) and the best performing reference GBM model (AUC = 0.767, AP = 0.334). The inclusion of imaging data and clinical variables in an ensemble model allowed statistically more powerful prediction of PFOA progression (AUC = 0.865, AP = 0.447), although the clinical significance of this minor performance gain remains unknown. The spatial attention module improved the predictive performance of the backbone model, and the visual interpretation of attention maps focused on the joint space and the regions where osteophytes typically occur. This study demonstrated the potential of machine learning models to predict the progression of PFOA using imaging and clinical variables. These models could be used to identify patients who are at high risk of progression and prioritize them for new treatments. However, even though the accuracy of the models were excellent in this study using the MOST dataset, they should be still validated using external patient cohorts in the future.

Which images to label for few-shot medical image analysis?

The success of deep learning methodologies hinges upon the availability of meticulously labeled extensive datasets. However, when dealing with medical images, the annotation process for such abundant training data often necessitates the involvement of experienced radiologists, thereby consuming their limited time resources. In order to alleviate this burden, few-shot learning approaches have been developed, which manage to achieve competitive performance levels with only several labeled images. Nevertheless, a crucial yet previously overlooked problem in few-shot learning is about the selection of template images for annotation before learning, which affects the final performance. In this study, we propose a novel TEmplate Choosing Policy (TECP) that aims to identify and select “the most worthy” images for annotation, particularly within the context of multiple few-shot medical tasks, including landmark detection, anatomy detection, and anatomy segmentation. TECP is composed of four integral components: (1) Self-supervised training, which entails training a pre-existing deep model to extract salient features from radiological images; (2) Alternative proposals for localizing informative regions within the images; and (3) Representative Score Estimation, which involves the evaluation and identification of the most representative samples or templates. (4) Ranking, which rank all candidates and select one with highest representative score. The efficacy of the TECP approach is demonstrated through a series of comprehensive experiments conducted on multiple public datasets. Across all three medical tasks, the utilization of TECP yields noticeable improvements in model performance.

Contagious infection-free medical interaction system with machine vision controlled by remote hand gesture during an operation

Background and objectiveMedical image visualization is a requirement in many types of surgery such as orthopaedic, spinal, thoracic procedures or tumour resection to eliminate risk such as “wrong level surgery”. However, direct contact with physical devices such as mice or touch screens to control images is a challenge because of the potential risk of infection. To prevent the spread of infection in sterile environments, a contagious infection-free medical interaction system has been developed for manipulating medical images. MethodsWe proposed an integrated system with three key modules: hand landmark detection, hand pointing, and hand gesture recognition. A proposed depth enhancement algorithm is combined with a deep learning hand landmark detector to generate hand landmarks. Based on the designed system, a proposed hand-pointing system combined with projection and ray-pointing techniques allows for reducing fatigue during manipulation. A proposed landmark geometry constraint algorithm and deep learning method were applied to detect six gestures including click, open, close, zoom, drag, and rotation. Additionally, a control menu was developed to effectively activate common functions. ResultsThe proposed hand-pointing system allowed for a large control range of up to 1200 mm in both vertical and horizontal direction. The proposed hand gesture recognition method showed high accuracy of over 97% and real-time response. ConclusionThis paper described the contagious infection-free medical interaction system that enables precise and effective manipulation of medical images within the large control range, while minimizing hand fatigue.

Deep Learning Method for Precise Landmark Identification and Structural Assessment of Whole-Spine Radiographs.

This study measured parameters automatically by marking the point for measuring each parameter on whole-spine radiographs. Between January 2020 and December 2021, 1017 sequential lateral whole-spine radiographs were retrospectively obtained. Of these, 819 and 198 were used for training and testing the performance of the landmark detection model, respectively. To objectively evaluate the program's performance, 690 whole-spine radiographs from four other institutions were used for external validation. The combined dataset comprised radiographs from 857 female and 850 male patients (average age 42.2 ± 27.3 years; range 20-85 years). The landmark localizer showed the highest accuracy in identifying cervical landmarks (median error 1.5-2.4 mm), followed by lumbosacral landmarks (median error 2.1-3.0 mm). However, thoracic landmarks displayed larger localization errors (median 2.4-4.3 mm), indicating slightly reduced precision compared with the cervical and lumbosacral regions. The agreement between the deep learning model and two experts was good to excellent, with intraclass correlation coefficient values >0.88. The deep learning model also performed well on the external validation set. There were no statistical differences between datasets in all parameters, suggesting that the performance of the artificial intelligence model created was excellent. The proposed automatic alignment analysis system identified anatomical landmarks and positions of the spine with high precision and generated various radiograph imaging parameters that had a good correlation with manual measurements.

DRIVER DROWSINESS DETECTION SYSTEM

The proposed system utilizes facial landmark detection and eye aspect ratio calculation to detect drowsiness in real-time from a webcam feed. It employs the dlib library for face detection and facial landmark prediction. The eye aspect ratio (EAR) is computed from the distances between specific points on the eyes. When the EAR falls below a predefined threshold for a certain number of consecutive frames, it triggers an alert indicating potential drowsiness. The alert includes visual cues on the video feed, an auditory alert using the playsound library, and even sends an instant WhatsApp message using pywhatkit to notify a designated contact about the drowsiness. This system serves as a useful tool for monitoring driver fatigue or alertness in various contexts, promoting safety and awareness. Its capacity to swiftly identify and alert users to the onset of drowsiness represents a significant leap forward in safety technology. By mitigating the risk of accidents stemming from fatigue-induced impairments, this project underscores the pivotal role of computer vision in real-time human behavior monitoring, offering a versatile and invaluable tool for fostering safety and awareness across various domains. Keywords: Drowsiness Detection; Python; Whatsapp messaging; Webcam feed; Computer Vision; Alert System; Real-time monitoring; Eye-Aspect Ratio; Frame Processing; Facial Recognition.