AbstractAugmented reality (AR) is gaining traction in the field of computer‐assisted treatment (CAT). Head‐mounted display (HMD)‐based AR in CAT provides dentists with enhanced visualisation by directly overlaying a three‐dimensional (3D) model on a real patient during dental treatment. However, conventional AR‐based treatments rely on optical markers and trackers, which makes them tedious, expensive, and uncomfortable for dentists. Therefore, a markerless image‐to‐patient tracking system is necessary to overcome these challenges and enhance system efficiency. This paper proposes a novel feature‐based markerless calibration and navigation method for an HMD‐based AR visualisation system. The authors address three sub‐challenges: firstly, synthetic RGB‐D data for anatomical landmark detection is generated to train a deep convolutional neural network (DCNN); secondly, the HMD is automatically calibrated using detected anatomical landmarks, eliminating the need for user input or optical trackers; and thirdly, a multi‐iterative closest point (ICP) algorithm is developed for effective 3D‐3D real‐time navigation. The authors conduct several experiments on a commercially available HMD (HoloLens 2). Finally, the authors compare and evaluate the approach against state‐of‐the‐art methods that employ HoloLens. The proposed method achieves a calibration virtual‐to‐real re‐projection distance of (1.09 ± 0.23) mm and navigation projection errors and accuracies of approximately (0.53 ± 0.19) mm and 93.87%, respectively.
Read full abstract