Landmark Information Research Articles

A vessel bifurcation liver CT landmark pair dataset for evaluating deformable image registration algorithms.

Evaluating deformable image registration (DIR) algorithms is vital for enhancing algorithm performance and gaining clinical acceptance. However, there is a notable lack of dependable DIR benchmark datasets for assessing DIR performance except for lung images. To address this gap, we aim to introduce our comprehensive liver computed tomography (CT) DIR landmark dataset library. This dataset is designed for efficient and quantitative evaluation of various DIR methods for liver CTs, paving the way for more accurate and reliable image registration techniques. Forty CT liver image pairs were acquired from several publicly available image archives and authors' institutions under institutional review board (IRB) approval. The images were processed with a semi-automatic procedure to generate landmark pairs: (1) for each case, liver vessels were automatically segmented on one image; (2) landmarks were automatically detected at vessel bifurcations; (3) corresponding landmarks in the second image were placed using two deformable image registration methods to avoid algorithm-specific biases; (4) a comprehensive validation process based on quantitative evaluation and manual assessment was applied to reject outliers and ensure the landmarks' positional accuracy. This workflow resulted in an average of ∼56 landmark pairs per image pair, comprising a total of 2220 landmarks for 40 cases. The general landmarking accuracy of this procedure was evaluated using digital phantoms and manual landmark placement. The landmark pair target registration errors (TRE) on digital phantoms were 0.37±0.26 and 0.55±0.34mm respectively for the two selected DIR algorithms used in our workflow, with 97% of landmark pairs having TREs below 1.5mm. The distances from the calculated landmarks to the averaged manual placement were 1.27±0.79mm. All data, including image files and landmark information, are publicly available at Zenodo (https://zenodo.org/records/13738577). Instructions for using our data can be found on our GitHub page at https://github.com/deshanyang/Liver-DIR-QA. The landmark dataset generated in this work is the first collection of large-scale liver CT DIR landmarks prepared on real patient images. This dataset can provide researchers with a dense set of ground truth benchmarks for the quantitative evaluation of DIR algorithms within the liver.

LiPro: displaying concurrent image viewing and imperceptible positioning using visible light

Visible light can be used to achieve effective human interaction with projectors. However, current visible light-based projectors for the target positioning either lose the image viewing function or cause abnormal luminance changes perceived by human eyes, which is not practical for real-world applications. Therefore, we propose a smart lighting-based projector system, LiPro, which can display imperceptible target positioning and any image viewing, simultaneously. In a nutshell, we propose what we believe to be a novel color space conversion modulation scheme to transmit the landmark information, which adds δ and −3δ color intensity value on the red (R) channel and blue (B) channel according to the YCbCr color space analysis, so that human eyes are imperceptible to light intensity changes. Then, we propose the dual-channel decision demodulation scheme, which demodulates the received light signals on both the R channel and B channel at the same time to decide and increase the correctness of the landmark recognition. In the end, we use off-the-shelf devices to implement LiPro to evaluate its performance, and results from extensive experiments show that our LiPro can achieve centimeter-level target positioning under any image viewing conditions.

Learning to use landmarks for navigation amplifies their representation in retrosplenial cortex.

Visual landmarks provide powerful reference signals for efficient navigation by altering the activity of spatially tuned neurons, such as place cells, head direction cells, and grid cells. To understand the neural mechanism by which landmarks exert such strong influence, it is necessary to identify how these visual features gain spatial meaning. In this study, we characterized visual landmark representations in mouse retrosplenial cortex (RSC) using chronic two-photon imaging of the same neuronal ensembles over the course of spatial learning. We found a pronounced increase in landmark-referenced activity in RSC neurons that, once established, remained stable across days. Changing behavioral context by uncoupling treadmill motion from visual feedback systematically altered neuronal responses associated with the coherence between visual scene flow speed and self-motion. To explore potential underlying mechanisms, we modeled how burst firing, mediated by supralinear somatodendritic interactions, could efficiently mediate context- and coherence-dependent integration of landmark information. Our results show that visual encoding shifts to landmark-referenced and context-dependent codes as these cues take on spatial meaning during learning.

Comparison of T1-weighted landmark placement and ROI transfer onto diffusion-weighted EPI sequences for targeted tractography tasks in the optic nerve.

Diffusion-based tractography in the optic nerve requires sampling strategies assisted by anatomical landmark information (regions of interest [ROIs]). We aimed to investigate the feasibility of expert-placed, high-resolution T1-weighted ROI-data transfer onto lower spatial resolution diffusion-weighted images. Slab volumes from 20 volunteers were acquired and preprocessed including distortion bias correction and artifact reduction. Constrained spherical deconvolution was used to generate a directional diffusion information grid (fibre orientation distribution-model [FOD]). Three neuroradiologists marked landmarks on both diffusion imaging variants and structural datasets. Structural ROI information (volumetric interpolated breath-hold sequence [VIBE]) was respectively registered (linear with 6/12 degrees of freedom [DOF]) onto single-shot EPI (ss-EPI) and readout-segmented EPI (rs-EPI) volumes, respectively. All eight ROI/FOD-combinations were compared in a targeted tractography task of the optic nerve pathway. Inter-rater reliability for placed ROIs among experts was highest in VIBE images (lower confidence interval 0.84 to 0.97, mean 0.91) and lower in both ss-EPI (0.61 to 0.95, mean 0.79) and rs-EPI (0.59 to 0.86, mean 0.70). Tractography success rate based on streamline selection performance was highest in VIBE-drawn ROIs registered (6-DOF) onto rs-EPI FOD (70.0% over 5%-threshold, capped to failed ratio 39/16) followed by both 12-DOF-registered (67.5%; 41/16) and nonregistered VIBE (67.5%; 40/23). On ss-EPI FOD, VIBE-ROI-datasets obtained fewer streamlines overall with each at 55.0% above 5%-threshold and with lower capped to failed ratio (6-DOF: 35/36; 12-DOF: 34/34, nonregistered 33/36). The combination of VIBE-placed ROIs (highest inter-rater reliability) with 6-DOF registration onto rs-EPI targets (best streamline selection performance) is most suitable for white matter template generation required in group studies.

Not seeing the forest for the trees: combination of path integration and landmark cues in human virtual navigation.

In order to successfully move from place to place, our brain often combines sensory inputs from various sources by dynamically weighting spatial cues according to their reliability and relevance for a given task. Two of the most important cues in navigation are the spatial arrangement of landmarks in the environment, and the continuous path integration of travelled distances and changes in direction. Several studies have shown that Bayesian integration of cues provides a good explanation for navigation in environments dominated by small numbers of easily identifiable landmarks. However, it remains largely unclear how cues are combined in more complex environments. To investigate how humans process and combine landmarks and path integration in complex environments, we conducted a series of triangle completion experiments in virtual reality, in which we varied the number of landmarks from an open steppe to a dense forest, thus going beyond the spatially simple environments that have been studied in the past. We analysed spatial behaviour at both the population and individual level with linear regression models and developed a computational model, based on maximum likelihood estimation (MLE), to infer the underlying combination of cues. Overall homing performance was optimal in an environment containing three landmarks arranged around the goal location. With more than three landmarks, individual differences between participants in the use of cues are striking. For some, the addition of landmarks does not worsen their performance, whereas for others it seems to impair their use of landmark information. It appears that navigation success in complex environments depends on the ability to identify the correct clearing around the goal location, suggesting that some participants may not be able to see the forest for the trees.

Memory consolidation affects the interplay of place and response navigation

Navigation through space is based on memory representations of landmarks (‘place’) or movement sequences (‘response’). Over time, memory representations transform through consolidation. However, it is unclear how the transformation affects place and response navigation in humans. In the present study, healthy adults navigated to target locations in a virtual maze. The preference for using place and response strategies and the ability to recall place and response memories were tested after a delay of one hour (n = 31), one day (n = 30), or two weeks (n = 32). The different delays captured early-phase synaptic changes, changes after one night of sleep, and long-delay changes due to the reorganization of navigation networks. Our results show that the relative contributions of place and response navigation changed as a function of time. After a short delay of up to one day, participants preferentially used a place strategy and exhibited a high degree of visual landmark exploration. After a longer delay of two weeks, place strategy use decreased significantly. Participants now equally relied on place and response strategy use and increasingly repeated previously taken paths. Further analyses indicate that response strategy use predominantly occurred as a compensatory strategy in the absence of sufficient place memory. Over time, place memory faded before response memory. We suggest that the observed shift from place to response navigation is context-dependent since detailed landmark information, which strongly relied on hippocampal function, decayed faster than sequence information, which required less detail and depended on extra-hippocampal areas. We conclude that changes in place and response navigation likely reflect the reorganization of navigation networks during systems consolidation.

A keypoints-motion-based landmark transfer method for face reenactment

Face reenactment is a task to transfer the facial pose and expression from one driving face to the source face. It can be done by adding many condition information, such as the landmark, 3D Morphable Model (3DMM). Because the landmark information is easy to get and contains face structure and expression information, many works rely on it. One of the key points for the landmark-based method is to generate landmarks that keep the face shape of the source image while having the same expression and pose as the driving image. Concentrating on this problem, we propose a novel and effective method to transfer the expression and pose based on the motion of the key points between driving images. Besides, we also try to improve the fit ability of the current popular backbone by adding an enhancement module.

Fixation-related potentials during mobile map assisted navigation in the real world: The effect of landmark visualization style.

An often-proposed enhancement for mobile maps to aid assisted navigation is the presentation of landmark information, yet understanding of the manner in which they should be displayed is limited. In this study, we investigated whether the visualization of landmarks as 3D map symbols with either an abstract or realistic style influenced the subsequent processing of those landmarks during route navigation. We utilized a real-world mobile electroencephalography approach to this question by combining several tools developed to overcome the challenges typically encountered in real-world neuroscience research. We coregistered eye-movement and EEG recordings from 45 participants as they navigated through a real-world environment using a mobile map. Analyses of fixation event-related potentials revealed that the amplitude of the parietal P200 component was enhanced when participants fixated landmarks in the real world that were visualized on the mobile map in a realistic style, and that frontal P200 latencies were prolonged for landmarks depicted in either a realistic or abstract style compared with features of the environment that were not presented on the map, but only for the male participants. In contrast, we did not observe any significant effects of landmark visualization style on visual P1-N1 peaks or the parietal late positive component. Overall, the findings indicate that the cognitive matching process between landmarks seen in the environment and those previously seen on a map is facilitated by more realistic map display, while low-level perceptual processing of landmarks and recall of associated information are unaffected by map visualization style.

Environment geometry alters subiculum boundary vector cell receptive fields in adulthood and early development

Boundaries to movement form a specific class of landmark information used for navigation: Boundary Vector Cells (BVCs) are neurons which encode an animal’s location as a vector displacement from boundaries. Here we characterise the prevalence and spatial tuning of subiculum BVCs in adult and developing male rats, and investigate the relationship between BVC spatial firing and boundary geometry. BVC directional tunings align with environment walls in squares, but are uniformly distributed in circles, demonstrating that environmental geometry alters BVC receptive fields. Inserted barriers uncover both excitatory and inhibitory components to BVC receptive fields, demonstrating that inhibitory inputs contribute to BVC field formation. During post-natal development, subiculum BVCs mature slowly, contrasting with the earlier maturation of boundary-responsive cells in upstream Entorhinal Cortex. However, Subiculum and Entorhinal BVC receptive fields are altered by boundary geometry as early as tested, suggesting this is an inherent feature of the hippocampal representation of space.

A comprehensive multimodal dataset for contactless lip reading and acoustic analysis

Small-scale motion detection using non-invasive remote sensing techniques has recently garnered significant interest in the field of speech recognition. Our dataset paper aims to facilitate the enhancement and restoration of speech information from diverse data sources for speakers. In this paper, we introduce a novel multimodal dataset based on Radio Frequency, visual, text, audio, laser and lip landmark information, also called RVTALL. Specifically, the dataset consists of 7.5 GHz Channel Impulse Response (CIR) data from ultra-wideband (UWB) radars, 77 GHz frequency modulated continuous wave (FMCW) data from millimeter wave (mmWave) radar, visual and audio information, lip landmarks and laser data, offering a unique multimodal approach to speech recognition research. Meanwhile, a depth camera is adopted to record the landmarks of the subject’s lip and voice. Approximately 400 minutes of annotated speech profiles are provided, which are collected from 20 participants speaking 5 vowels, 15 words, and 16 sentences. The dataset has been validated and has potential for the investigation of lip reading and multimodal speech recognition.

Memory effects of visual and olfactory landmark information in human wayfinding

Non-human animals are exceptionally good at using smell to find their way through the environment. However, the use of olfactory cues for human navigation is often underestimated. Although the sense of smell is well-known for its distinct connection to memory and emotion, memory effects in human navigation using olfactory landmarks have not been studied yet. Therefore, this article compares wayfinding and recognition performance for visual and olfactory landmarks learned by 52 participants in a virtual maze. Furthermore, it is one of the first empirical studies investigating differences in memory effects on human navigation by using two separate test situations 1 month apart. The experimental task was to find the way through a maze-like virtual environment with either olfactory or visual cues at the intersections that served as decision points. Our descriptive results show that performance was above chance level for both conditions (visual and olfactory landmarks). Wayfinding performance did not decrease 1 month later when using olfactory landmarks. In contrast, when using visual landmarks wayfinding performance decreased significantly, while visual landmarks overall lead to better recognition than olfactory landmarks at both times of testing. The results demonstrate the unique character of human odor memory and support the conclusion that olfactory cues may be used in human spatial orientation. Furthermore, the present study expands the research field of human wayfinding by providing a study that investigates memory for landmark knowledge and route decisions for the visual and olfactory modality. However, more studies are required to put this important research strand forward.

Implicit versus explicit processing of visual, olfactory, and multimodal landmark information in human wayfinding.

Despite the predominant focus on visual perception in most studies, the role of humans' sense of smell in navigation has often been neglected. Recent research, however, could show that humans are indeed able to use their sense of smell for orientation, particularly when processed implicitly. In this study, we investigate whether implicit perception of olfactory landmarks enhanced wayfinding performance compared to explicit perception. Fifty-two people completed a wayfinding and a recognition task in a virtual maze at two times of testing 1 month apart. Participants either received olfactory, visual, or both cues at the intersections. Wayfinding performance was better for olfactory landmarks, which were not correctly remembered in the recognition task. In contrast, wayfinding performance was better when visual landmarks were correctly remembered. In the multimodal condition, wayfinding performance was better with landmarks being remembered at t1 and remained the same at t2. Our results suggest distinct implicit processing mechanisms within the olfactory system and therefore hold important implications for the nature of spatial odor processing extending beyond explicit odor localization tasks. The study highlights the importance for future studies to develop and employ further experimental methods that capture implicit processing across all of our senses. This is crucial for a comprehensive understanding of consciousness, as olfaction strongly influences our behavior, but remains largely latent unless deliberately honed through practice.

Cascading CNNs for facial action unit detection

The contractions of facial muscles are what shape the expressions produced by the human face. The Facial Action Coding System (FACS) stands as the predominant standard in describing all visual alterations in the face, defining them through Action Units (AU) that articulate the movements occurring in the facial muscles. In this paper, an end-to-end pipeline, CCNN2, is proposed as a deep pre-processing step to detect AUs by processing the features extracted from hidden CNN layers, without exploiting any landmark information in a recursive manner. Trials conducted on three spontaneous datasets (MMI, DISFA, BP4D) along with one in-the-wild dataset (EmotioNet) demonstrate that this method surpasses the results of state-of-the-art approaches in three of the datasets, and even more, its two-module structure increases the overall F1 score in detection in every experiment. The method being proposed is also adaptable to a diverse range of classification applications.

Integration of landmark and saccade target signals in macaque frontal cortex visual responses

Visual landmarks influence spatial cognition and behavior, but their influence on visual codes for action is poorly understood. Here, we test landmark influence on the visual response to saccade targets recorded from 312 frontal and 256 supplementary eye field neurons in rhesus macaques. Visual response fields are characterized by recording neural responses to various target-landmark combinations, and then we test against several candidate spatial models. Overall, frontal/supplementary eye fields response fields preferentially code either saccade targets (40%/40%) or landmarks (30%/4.5%) in gaze fixation-centered coordinates, but most cells show multiplexed target-landmark coding within intermediate reference frames (between fixation-centered and landmark-centered). Further, these coding schemes interact: neurons with near-equal target and landmark coding show the biggest shift from fixation-centered toward landmark-centered target coding. These data show that landmark information is preserved and influences target coding in prefrontal visual responses, likely to stabilize movement goals in the presence of noisy egocentric signals.

Olfactory and visual vs. multimodal landmark processing in human wayfinding: a virtual reality experiment

ABSTRACT Recent studies suggest that the olfactory system is not as indiscriminative as previously thought. Odours can serve as useful landmark information. The current study explored recognition and wayfinding performance in 54 participants (52 at t2), who either received 18 unimodal (only visual or olfactory) or multimodal (visual-olfactory) landmark information during a learning phase in a virtual reality maze before finding their way through it. Recognition performance was assessed in a recognition task (18 targets, 18 distractors). The experiment was repeated after one month. As expected, participants showed poorer recognition performance if presented with odours compared to the other two conditions. But, recognition performance did not decline over time. Wayfinding performance was similar across all conditions. However, odours were the only condition with stable performance over time, possibly due to emotional mediation and deeper processing. Multimodal material added no value to wayfinding performance, supposedly due to higher cognitive load.

Drop-relationship learning for semi-supervised facial action unit recognition

Facial action units (AUs) recognition becomes important for facial analysis and has been widely applied in psychological research. Existing work on AU relationship learning only considers the relationship between AUs simultaneously, and does not consider the complex co-adaptation between AUs. Co-adaptation is good, but not always good. Learning the AU relationship simultaneously will inhibit the robust features of network learning. And AU relationship learning often requires additional landmark information. To solve these problems, we propose a novel AU Drop-relationship learning inspired by dropout. The AU Drop-relationship learning constructs the AU relationship units through prior knowledge. We further design the relationship regularization module to constraint the relationship between AU pairs. We randomly drop AU relationship units during training to suppress co-adaptation, forcing the network to learn more robust features. In addition, considering that there are massive unlabeled web facial images in reality, manual labeling of AU requires experts and is particularly time-consuming. To address this problem, we propose a method composed of consistency regularization and pseudo-multi-labeling for semi-supervised AU recognition. The proposed method outperforms the state-of-the-art semi-supervised and supervised methods on two widely used AU datasets (BP4D and DISFA).

Development of a cross-artificial intelligence system for identifying intraoperative anatomical landmarks and surgical phases during laparoscopic cholecystectomy.

Attention to anatomical landmarks in the appropriate surgical phase is important to prevent bile duct injury (BDI) during laparoscopic cholecystectomy (LC). Therefore, we created a cross-AI system that works with two different AI algorithms simultaneously, landmark detection and phase recognition. We assessed whether landmark detection was activated in the appropriate phase by phase recognition during LC and the potential contribution of the cross-AI system in preventing BDI through a clinical feasibility study (J-SUMMIT-C-02). A prototype was designed to display landmarks during the preparation phase and Calot's triangle dissection. A prospective clinical feasibility study using the cross-AI system was performed in 20 LC cases. The primary endpoint of this study was the appropriateness of the detection timing of landmarks, which was assessed by an external evaluation committee (EEC). The secondary endpoint was the correctness of landmark detection and the contribution of cross-AI in preventing BDI, which were assessed based on the annotation and 4-point rubric questionnaire. Cross-AI-detected landmarks in 92% of the phases where the EEC considered landmarks necessary. In the questionnaire, each landmark detected by AI had high accuracy, especially the landmarks of the common bile duct and cystic duct, which were assessed at 3.78 and 3.67, respectively. In addition, the contribution to preventing BDI was relatively high at 3.65. The cross-AI system provided landmark detection at appropriate situations. The surgeons who previewed the model suggested that the landmark information provided by the cross-AI system may be effective in preventing BDI. Therefore, it is suggested that our system could help prevent BDI in practice. Trial registration University Hospital Medical Information Network Research Center Clinical Trial Registration System (UMIN000045731).

A facial geometry based detection model for face manipulation using CNN-LSTM architecture

This issue of DeepFake technique that may cause great threat to privacy, democracy, and national security has attracted the attention of deep learning researchers. DeepFake detection, therefore, has been a very hot issue in deep learning research. The face landmark feature maps are often used by many DeepFake approaches in generating fake faces. It also provides key information to help to detect manipulated face images. In this paper, we propose a detection approach for manipulated face images. To make full use of face landmark information, we propose a facial geometry prior module (FGPM) to extract facial geometry feature maps. Then the facial geometry feature maps are embedded into upsampling feature maps generated by the CNN-LSTM network. The proposed FGPM exploits facial maps and frequency domain correlation to analyze the discriminative characteristics between manipulated and non-manipulated regions by incorporating the CNN-LSTM network. Finally, a decoder is used to learn the mapping from low-resolution feature maps to pixel-wise to predict manipulation localization. Or a softmax classifier is used to predict real or fake face images. By experimenting on several popular datasets, the proposed detection model has demonstrated the capability of localizing manipulation at the pixel level and with a high prediction on real or fake face images.

Analysis and Visualization of Vessels’ RElative MOtion (REMO)

This research is a pilot study to develop a maritime traffic control system that supports the decision-making process of control officers, and to evaluate the usability of a prototype tool developed in this study. The study analyzed the movements of multiple vessels through automatic identification system (AIS) data using one of the existing methodologies in GIScience, the RElative MOtion (REMO) approach. The REMO approach in this study measured the relative speed, delta-speed, and the azimuth of each vessel per time unit. The study visualized the results on electronic navigational charts in the prototype tool developed, V-REMO. In addition, the study conducted a user evaluation to assess the user interface (UI) of V-REMO and to future enhance the usability. The general usability of V-REMO, the data visualization, and the readability of information in the UI were tested through in-depth interviews. The results of the user evaluation showed that the users needed changes in the size, position, colors, and transparency of the trajectory symbols in the digital chartmap view of V-REMO for better readability and easier manipulation. The users also indicated a need for multiple color schemes for the spatial data and more landmark information about the study area in the chartmap view.

Online facial expression recognition based on graph convolution and long short memory networks

Video‐based facial expression recognition (FER) models have achieved higher accuracy with more computation, which is not suitable for online deployment in mobile intelligent terminals. Facial landmarks can model facial expression changes with their spatial location information instead of texture features. But classical convolution operation cannot make full use of landmark information. To this end, in this paper, we propose a novel long short memory network (LSTM) by embedding graph convolution named GELSTM for online video‐based FER in mobile intelligent terminals. Specifically, we construct landmark‐based face graph data from the client. On the server side, we introduce graph convolution which can effectively mine spatial dependencies information in a landmark‐based facial graph. Moreover, the extracted landmark's features are fed to LSTM for temporal feature aggregation. We conduct experiments on the facial expression dataset and the results show our proposed method shows superior performance compared to other deep models.