Navigation Instructions Research Articles

Wireless programmable patterns of electro-hydraulic haptic electronic skins able to create surface morphology

Haptic feedback technology represents a frontier with revolutionary potential, particularly in virtual and augmented reality. Skin, as an underexplored sensory interface, holds tremendous potential to significantly enrich our perceptual experiences while directly impacting the fields of communication, entertainment, and medicine. This article presents a programmable, ultra-lightweight, and ultra-thin wireless wearable haptic electronic skin that achieves rapid response and large mechanical feedback force and deformation displacement output. The electronic skin comprises a series of haptic electro-hydraulic actuators, each with the weight of 0.023 g and the thickness of 199 μm. Each actuator can be independently regulated and produce a maximum normal force of 150 mN and a normal displacement of 0.58 mm, simultaneously achieving the haptic presentation of textures. Relying on a portable multi-channel control circuit system designed, each electro-hydraulic actuator of the haptic electronic skin can be precisely control to achieve various haptic patterns. Showcasing its potentials, the haptic technology aids visual-impaired individuals in reading books, provides ground navigation instructions, and enhances gaming experiences, contributing to a more immersive and authentic artificial intelligence environment.

Snap&Nav: Smartphone-based Indoor Navigation System For Blind People via Floor Map Analysis and Intersection Detection

We present Snap&Nav, a navigation system for blind people in unfamiliar buildings, without prebuilt digital maps. Instead, the system utilizes the floor map as its primary information source for route guidance. The system requires a sighted assistant to capture an image of the floor map, which is analyzed to create a node map containing intersections, destinations, and current positions on the floor. The system provides turn-by-turn navigation instructions while tracking users' positions on the node map by detecting intersections. Additionally, the system estimates the scale difference of the node map to provide distance information. Our system was validated through two user studies with 20 sighted and 12 blind participants. Results showed that sighted participants processed floor map images without being accustomed to the system, while blind participants navigated with increased confidence and lower cognitive load compared to the condition using only cane, appreciating the system's potential for use in various buildings.

Enhancing Indoor Navigation for Visually Impaired Individuals with an AI Chatbot Utilizing VEO Optimized Nodes and Natural Language Processing

Aims and Background: Visually impaired people face numerous challenges when it comes to indoor navigation. While outdoor navigation benefits from advancements in GPS and related technologies, indoor spaces present intricate, complex, and often less accessible environments for those with visual impairments. Objective and Methodology: In response to these challenges, we propose an innovative approach to enhance indoor navigation for individuals with visual impairments, leveraging the power of an AI chatbot. Our AI chatbot employs cutting-edge artificial intelligence techniques to provide realtime assistance and guidance, facilitating independent navigation within intricate indoor settings. By harnessing natural language processing technologies, the chatbot engages in intuitive interactions with users, comprehending their queries and offering detailed instructions for efficient indoor navigation. The main goal of this research is to enhance the independence of people with visual impairments by offering them a reliable and easily accessible tool. Results and conclusion: This tool, driven by our Volcano Eruption Optimization Network, promises to significantly enhance the independence and overall indoor navigation experience for visually impaired people, ultimately fostering a greater sense of autonomy in navigating complex indoor spaces.

Autonomous Vision-Based Navigation and Stability Augmentation Control of a Biomimetic Robotic Hammerhead Shark

The application potential of robotic fish embedded with intelligent visual navigation algorithms in underwater autonomous operation is on full display recently. However, the existing visual navigation methods are limited by the underwater visual conditions and the motion characteristics of robotic fish. To this end, this paper proposes a novel autonomous navigation framework integrated with visual stabilization control. In practice, a stereo vision-based navigation network is proposed to generate the guidance law. On this basis, a biomimetic robotic hammerhead shark with a controllable cephalofoil is developed, and a nonlinear model predictive controller for cephalofoil stabilization relying on the dynamic model is elaborately designed. Extensive simulations and underwater experiments are conducted to validate the effectiveness and superiority of the proposed methods, which significantly enhance exploration efficiency and reduce image jitter by 26.02% compared to the traditional methods. The obtained results provide a new idea for underwater robots to autonomously explore the ocean. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper is motivated by the problem of vision-based underwater autonomous navigation for a biomimetic robotic fish that possesses underwater visual stability and good maneuverability. The existing visual navigation networks usually generate unexpected navigation instructions when dealing with complex or ambiguous underwater scenes. Additionally, image jitter caused by the rhythmic motion of robotic fish can lead to navigation failure. This paper suggests an integrated navigation framework that includes a biomimetic platform design, a visual stabilization controller, and an intelligent underwater navigation network. Specifically, a novel sphyrnidae-inspired robotic shark is designed as a new platform with superior motion performance. To enhance underwater visual stability, a nonlinear model predictive control-based visual stabilization controller is proposed. Furthermore, a deep stereo attention navigation network based on a parallax attention mechanism is proposed to improve the generalization of vision-based autonomous navigation. A series of underwater search experiments on the robotic shark demonstrate the effectiveness and superiority of the proposed navigation framework. Hopefully, our proposed methods can provide valuable guidance and support for universal underwater robot navigation to accomplish practical marine tasks, such as underwater rescue, resource exploitation, biological observation, and so on.

Learning to Follow and Generate Instructions for Language-Capable Navigation

Learning to Follow and Generate Instructions for Language-Capable Navigation

VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View

Incremental decision making in real-world environments is one of the most challenging tasks in embodied artificial intelligence. One particularly demanding scenario is Vision and Language Navigation (VLN) which requires visual and natural language understanding as well as spatial and temporal reasoning capabilities. The embodied agent needs to ground its understanding of navigation instructions in observations of a real-world environment like Street View. Despite the impressive results of LLMs in other research areas, it is an ongoing problem of how to best connect them with an interactive visual environment. In this work, we propose VELMA, an embodied LLM agent that uses a verbalization of the trajectory and of visual environment observations as contextual prompt for the next action. Visual information is verbalized by a pipeline that extracts landmarks from the human written navigation instructions and uses CLIP to determine their visibility in the current panorama view. We show that VELMA is able to successfully follow navigation instructions in Street View with only two in-context examples. We further finetune the LLM agent on a few thousand examples and achieve around 25% relative improvement in task completion over the previous state-of-the-art for two datasets.

VLN-Video: Utilizing Driving Videos for Outdoor Vision-and-Language Navigation

Outdoor Vision-and-Language Navigation (VLN) requires an agent to navigate through realistic 3D outdoor environments based on natural language instructions. The performance of existing VLN methods is limited by insufficient diversity in navigation environments and limited training data. To address these issues, we propose VLN-Video, which utilizes the diverse outdoor environments present in driving videos in multiple cities in the U.S. augmented with automatically generated navigation instructions and actions to improve outdoor VLN performance. VLN-Video combines the best of intuitive classical approaches and modern deep learning techniques, using template infilling to generate grounded non-repetitive navigation instructions, combined with an image rotation similarity based navigation action predictor to obtain VLN style data from driving videos for pretraining deep learning VLN models. We pre-train the model on the Touchdown dataset and our video-augmented dataset created from driving videos with three proxy tasks: Masked Language Modeling, Instruction and Trajectory Matching, and Next Action Prediction, so as to learn temporally-aware and visually-aligned instruction representations. The learned instruction representation is adapted to the state-of-the-art navigation agent when fine-tuning on the Touchdown dataset. Empirical results demonstrate that VLN-Video significantly outperforms previous state-of-the-art models by 2.1% in task completion rate, achieving a new state-of-the-art on the Touchdown dataset.

CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments

Audio-visual navigation of an agent towards locating an audio goal is a challenging task especially when the audio is sporadic or the environment is noisy. In this paper, we present CAVEN, a Conversation-based Audio-Visual Embodied Navigation framework in which the agent may interact with a human/oracle for solving the task of navigating to an audio goal. Specifically, CAVEN is modeled as a budget-aware partially observable semi-Markov decision process that implicitly learns the uncertainty in the audio-based navigation policy to decide when and how the agent may interact with the oracle. Our CAVEN agent can engage in fully-bidirectional natural language conversations by producing relevant questions and interpret free-form, potentially noisy responses from the oracle based on the audio-visual context. To enable such a capability, CAVEN is equipped with: i) a trajectory forecasting network that is grounded in audio-visual cues to produce a potential trajectory to the estimated goal, and (ii) a natural language based question generation and reasoning network to pose an interactive question to the oracle or interpret the oracle's response to produce navigation instructions. To train the interactive modules, we present a large scale dataset: AVN-Instruct, based on the Landmark-RxR dataset. To substantiate the usefulness of conversations, we present experiments on the benchmark audio-goal task using the SoundSpaces simulator under various noisy settings. Our results reveal that our fully-conversational approach leads to nearly an order-of-magnitude improvement in success rate, especially in localizing new sound sources and against methods that use only uni-directional interaction.

NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models

Trained with an unprecedented scale of data, large language models (LLMs) like ChatGPT and GPT-4 exhibit the emergence of significant reasoning abilities from model scaling. Such a trend underscored the potential of training LLMs with unlimited language data, advancing the development of a universal embodied agent. In this work, we introduce the NavGPT, a purely LLM-based instruction-following navigation agent, to reveal the reasoning capability of GPT models in complex embodied scenes by performing zero-shot sequential action prediction for vision-and-language navigation (VLN). At each step, NavGPT takes the textual descriptions of visual observations, navigation history, and future explorable directions as inputs to reason the agent's current status, and makes the decision to approach the target. Through comprehensive experiments, we demonstrate NavGPT can explicitly perform high-level planning for navigation, including decomposing instruction into sub-goals, integrating commonsense knowledge relevant to navigation task resolution, identifying landmarks from observed scenes, tracking navigation progress, and adapting to exceptions with plan adjustment. Furthermore, we show that LLMs is capable of generating high-quality navigational instructions from observations and actions along a path, as well as drawing accurate top-down metric trajectory given the agent's navigation history. Despite the performance of using NavGPT to zero-shot R2R tasks still falling short of trained models, we suggest adapting multi-modality inputs for LLMs to use as visual navigation agents and applying the explicit reasoning of LLMs to benefit learning-based models. Code is available at: https://github.com/GengzeZhou/NavGPT.

Enhancing wayfinding and route learning: a human-centred study of the route-defining locations algorithm

ABSTRACT Navigation services play a crucial role in everyday wayfinding. One of their key features is the turn-by-turn guidance, enabling wayfinders to navigate efficiently between locations. However, over-reliance on turn-by-turn guidance can hinder wayfinders' ability to perceive their surroundings and develop spatial awareness, negatively impacting user experience. Previous research has addressed these issues by modifying instructions, user interaction, or both. However, the applicability of these methods beyond specific studies remains unclear. This paper presents a human-centred study by evaluating an algorithm that automatically generates navigation instructions. The instructions are generated by adding route-defining locations to ‘standard’ turn-by-turn instructions to enhance wayfinder's spatial knowledge acquisition, and improve user experience. Route-defining locations, identified using a previously proposed algorithm, serve as anchors for the wayfinder to form a spatial understanding of the environment. The study involved 36 participants divided into two groups: one received ‘standard’ turn-by-turn (TBT) instructions, while the other received instructions that included route-defining locations (RDL). Participants navigated in an unfamiliar virtual environment, and their wayfinding and route learning were compared between the two groups. Results show that RDL instructions led to better wayfinding and route learning performance compared to TBT instructions, suggesting the potential to improve navigation systems and user experiences.

Visual-and-Language Multimodal Fusion for Sweeping Robot Navigation Based on CNN and GRU

Effectively fusing information between the visual and language modalities remains a significant challenge. To achieve deep integration of natural language and visual information, this research introduces a multimodal fusion neural network model, which combines visual information (RGB images and depth maps) with language information (natural language navigation instructions). Firstly, the authors used faster R-CNN and ResNet50 to extract image features and attention mechanism to further extract effective information. Secondly, GRU model is used to extract language features. Finally, another GRU model is used to fuse the visual- language features, and then the history information is retained to give the next action instruction to the robot. Experimental results demonstrate that the proposed method effectively addresses the localization and decision-making challenges for robotic vacuum cleaners.

Usability, Appeal, and Relevance of Music and Wellbeing Training for Carers of People with Dementia: A Think-Aloud Study.

Personalized music listening can be highly effective in supporting people experiencing changes in mood and behavior due to Alzheimer's disease and other conditions causing dementia. However, a lack of staff education can be a barrier to effective use of music in residential aged care. This study investigates the usability, appeal and relevance of an online training course designed to help care staff develop personalized music programs for people living with dementia in their care. A mixed methods approach was taken in which 13 participants took part in a think-aloud session while using the training. Qualitative data from the think-aloud session and interviews were triangulated with results of a survey evaluation of the training. Themes relating to Engagement, Usability and Appeal, Pedagogical Design and Content, and Transfer, Impact and Barriers to Uptake were discussed. Results indicated that participants regarded the virtual environment and pedagogical design as of high quality, although some minor issues in navigability were identified. Participants also found the training to be highly relevant to their roles as caregivers and reported ways they would incorporate key concepts into care practices. However, ongoing barriers to implementation of such training were identified including systemic issues that contribute to a culture of focus on physical care. Consistency of navigation and clear instructions for interacting with content are important in training of time-poor care staff. Linking training to industry standards can provide important motivation for implementation of training, although systemic barriers can impede real-world change.

Narrating the route: route memorability in navigation instructions augmented with narrative–results from a user study

ABSTRACT From oral histories to mnemonic devices, humans have an excellent ability to remember object sequences and their relationships inside of narratives. In pedestrian wayfinding, remembering landmarks and their relationships is considered key to learning routes. This research explores whether augmenting verbal route instructions with a narrative increases the memorability of a route. Narrative theory is applied as a framework to develop narrative-based navigation instructions, which were tested in a field study (N = 18). After learning a route, participants recalled the route verbally, completed a photo-based landmark sequencing task and discussed their answers. One week later, a route recognition task and a second photo-based landmark sequencing task was completed online. Results show few significant differences between the two groups when compared quantitatively. However, during interviews, the narrative group repeatedly cited the narrative when remembering the route. The results suggest that incorporating narratives into route directions can be further explored, and that some novel direction types may not be well-measured using quantitative methods. This research confirms the prowess of landmark-based instructions to facilitate route memory, contributes to the growing body of work augmenting landmark-based route directions with detailed information, and further encourages designers to consider alternate route communication methods.

Limiting the reliance on navigation assistance with navigation instructions containing emotionally salient narratives for confident wayfinding

We live in a world that is increasingly dependent on technology, including for orientation in both familiar and unfamiliar space, which contributes to a long-term erosion of innate spatial navigation skills. In this study, we examined whether modified navigation instructions can make pedestrians less reliant on navigation aids to solve wayfinding tasks. In contrast to standard instructions, the modified instructions make decision-relevant landmarks at intersections emotionally salient and connected through narrative, and thus more memorable. The results of our online VR study with seventy adults revealed that, after navigating an unfamiliar route using modified navigation instructions, people made significantly fewer references to the navigation aid without compromising the accuracy of navigation compared to standard instructions. Narrative-based navigation instructions improved memory for the order in which relevant features in the environment were encountered along the traversed route, but not landmark recognition memory or memory for landmark-direction associations. Our findings highlight the benefits of using human-centred technologies that – as opposed to current navigation systems – promote the encoding and memorability of spatial information during navigation, and have the potential to train human spatial navigation abilities in the long term as a countermeasure toward GPS cognitive deskilling of population.

Following Navigation Instructions: Interdependent Spatial and Verbal Working Memory Processes

Following Navigation Instructions: Interdependent Spatial and Verbal Working Memory Processes

Turn Left Turn Right - Delving type and modality of instructions in navigation assistant systems for people with visual impairments

Receiving navigation directions and relevant information through appropriate channels is crucial for individuals with visual impairments when they use navigation assistant systems. Various navigation assistant systems have implemented diverse methods and modalities to provide navigation instructions to users. Moreover, several studies have examined the preferences of individuals with visual impairments regarding modalities. These studies have primarily relied on surveys or interviews to report their findings. In this study, we extensively investigated the type of navigation information individuals with visual impairments would prefer to receive. To achieve this, we employed a mixed-method design comprising an experiment, a questionnaire, and interviews conducted with participants with visual impairments and individuals with 20/20 vision who were blindfolded. We tested unimodal instruction modes (audio only) and multimodal instruction modes (audio and vibration), along with two types of audio instructions: short and descriptive. To evaluate the participants’ navigation performance, we measured the time taken and errors made during navigation using different instruction types and modes. The findings of this study suggest that the instruction mode does not have any significant effect on the time taken or errors made to complete the trials. Additionally, no evidence was found to support the preference for descriptive instructions over short instructions among the participants. Furthermore, the study did not find evidence to suggest that participants prefer receiving detailed instructions about the environment and obstacles during navigation. It became evident that participants had varying preferences for instruction modes, highlighting the importance of customized functionalities in navigation assistant systems. Moreover, our study validated that participants with visual impairments spent less time and made fewer errors during navigation compared to blindfolded participants with 20/20 vision. We believe that the results of this study will contribute to expanding the understanding of user preferences regarding the type and modality of instructions in the domain of assistive navigation technology.

From Floorplan to Navigation Concepts: Automatic Generation of Text-based Games

Abstract. Text-based games are environments in which defining the world, the representation of the world to the player (hereafter, agent) and agent interactions with the environment are all through text. Text-based games expose abstract, executable representations of indoor spaces through verbally referenced concepts. Yet, the ability of text-based games to represent indoor environments of real-world complexity is currently limited due to insufficient support for complex space decomposition and space interaction concepts. This paper suggests a procedure to automate the mapping of real-world geometric floorplan information into text-based game environment concepts, using the Microsoft TextWorld game platform as a case. To capture the complexities of indoor spaces, we enrich existing TextWorld concepts supported by theoretical navigation concepts.We first decompose indoor spaces using skeletonization, and then identify formal space concepts and their relationships. We further enhance the spectrum of supported agent interactions with an extended grammar, including egocentric navigation instructions. We demonstrate and discuss these new capabilities in an evacuation scenario. Our implementation extends the capabilities of TextWorld to provide a research testbed for spatial research, including symbolic spatial modelling, interaction with indoor spaces, and agent-based machine learning and language processing tasks.

Analysis of Collision Hazard Prevention With Operating Radar NICD – 453 Type On MV Amrta Jaya 1

The navigation system is the key to the safety of the ship in carrying out the voyage. RADAR will be very useful during bad weather, foggy conditions and sailing at night, especially when navigational instructions such as beacons, buoys, hills or buildings cannot be visually observed. So prospective ship officers must understand and understand to use RADAR navigation tools. To find out the target position and reduce the accident rate in the shipping lane, the role of RADAR in supporting shipping safety is needed. This study aims to analyze the prevention of collision hazards by using the Maneuvering Board in the operation of the Binocular RADAR at MV AMRTA JAYA 1. The type of research used in this study is quantitative research. In this case the data collection techniques used are documentation and observation. Documentation is in the form of searching for information in the form of books and documents, while observation is in the form of an approach to the object of research. From the results of research conducted, it can be seen that the method used to calculate the RADAR Plotting Sheet can be done by theoretical methods with formulas and scale methods with mathematical logic. The two methods can be used with the same results but with different time efficiency. The use of muvering boards and RADAR Plotting Sheet calculations can help identify the presence or absence of a collision hazard, and help estimate the closest distance, time taken, course and speed of the target ship.&#x0D;

Sonification of navigation instructions for people with visual impairment

Navigation assistance services for people with visual impairment pose the challenge of providing accurate guidance through non-visual navigation instructions. This paper proposes two techniques to guide the user during navigation assistance, and in particular during turns and straight paths. Both techniques adopt a combination of speech and non-speech audio with the aim of providing continuous, accurate, and unobtrusive guidance. The two solutions differ in the sonification technique used to generate the non-speech audio. The techniques were evaluated in a real-world environment with nine participants having severe visual impairment. Experimental results show that one of the proposed techniques is significantly more effective than a baseline approach adopted in existing solutions, in terms of navigation accuracy during turns and straight paths. Thus, it is a practical and effective solution to the problem of non-visual navigation assistance for people with visual impairment.

Evaluating Augmented Reality Landmark Cues and Frame of Reference Displays with Virtual Reality.

Daily travel usually demands navigation on foot across a variety of different application domains, including tasks like search and rescue or commuting. Head-mounted augmented reality (AR) displays provide a preview of future navigation systems on foot, but designing them is still an open problem. In this paper, we look at two choices that such AR systems can make for navigation: 1) whether to denote landmarks with AR cues and 2) how to convey navigation instructions. Specifically, instructions can be given via a head-referenced display (screen-fixed frame of reference) or by giving directions fixed to global positions in the world (world-fixed frame of reference). Given limitations with the tracking stability, field of view, and brightness of most currently available head-mounted AR displays for lengthy routes outdoors, we decided to simulate these conditions in virtual reality. In the current study, participants navigated an urban virtual environment and their spatial knowledge acquisition was assessed. We experimented with whether or not landmarks in the environment were cued, as well as how navigation instructions were displayed (i.e., via screen-fixed or world-fixed directions). We found that the world-fixed frame of reference resulted in better spatial learning when there were no landmarks cued; adding AR landmark cues marginally improved spatial learning in the screen-fixed condition. These benefits in learning were also correlated with participants' reported sense of direction. Our findings have implications for the design of future cognition-driven navigation systems.