Natural Language Commands Research Articles

Open-set 3D semantic instance maps for vision language navigation – O3D-SIM

Humans excel at forming mental maps of their surroundings, equipping them to understand object relationships and navigate based on language queries. Our previous work SI Maps (Nanwani L, Agarwal A, Jain K, et al. Instance-level semantic maps for vision language navigation. In: 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE; 2023 Aug.) showed that having instance-level information and the semantic understanding of an environment helps significantly improve performance for language-guided tasks. We extend this instance-level approach to 3D while increasing the pipeline's robustness and improving quantitative and qualitative results. Our method leverages foundational models for object recognition, image segmentation, and feature extraction. We propose a representation that results in a 3D point cloud map with instance-level embeddings, which bring in the semantic understanding that natural language commands can query. Quantitatively, the work improves upon the success rate of language-guided tasks. At the same time, we qualitatively observe the ability to identify instances more clearly and leverage the foundational models and language and image-aligned embeddings to identify objects that, otherwise, a closed-set approach wouldn't be able to identify. Project Page - https://smart-wheelchair-rrc.github.io/o3d-sim-webpage

Enhancing smart home interaction through multimodal command disambiguation

AbstractSmart speakers are entering our homes and enriching the connected ecosystem already present in them. Home inhabitants can use those to execute relatively simple commands, e.g., turning a lamp on. Their capabilities to interpret more complex and ambiguous commands (e.g., make this room warmer) are limited, if not absent. Large language models (LLMs) can offer creative and viable solutions to enable a practical and user-acceptable interpretation of such ambiguous commands. This paper introduces an interactive disambiguation approach that integrates visual and textual cues with natural language commands. After contextualizing the approach with a use case, we test it in an experiment where users are prompted to select the appropriate cue (an image or a textual description) to clarify ambiguous commands, thereby refining the accuracy of the system’s interpretations. Outcomes from the study indicate that the disambiguation system produces responses well-aligned with user intentions, and that participants found the textual descriptions slightly more effective. Finally, interviews reveal heightened satisfaction with the smart-home system when engaging with the proposed disambiguation approach.

An LLM-based vision and language cobot navigation approach for Human-centric Smart Manufacturing

Industry 5.0 prioritizes Human-centric Smart Manufacturing (HSM), aiming to enhance human operators’ well-being and needs. This necessitates collaborative robots with advanced natural interaction capabilities and improved perception, cognition, and action intelligence. The Large Language Model (LLM) exhibits strong reasoning abilities and generalization capabilities, which can significantly advance the development of HSM once integrated into human–robot interaction and collaboration. Accordingly, this paper explores part of LLM’s ability in the context of smart manufacturing, focusing on addressing interruptions in the manufacturing process caused by repetitive tool fetching. To alleviate this issue, a vision and language cobot navigation approach is innovatively adopted in the manufacturing environment that could be further used to assist operators in retrieving tools. Specifically, a real Human–Robot Collaboration (HRC) manufacturing scene is first reconstructed and annotated using Three-Dimensional (3D) point cloud techniques. Then the LLM is utilized for the Automated Guided Vehicle (AGV) to comprehend natural language commands and generate Python code to triger navigation actions. Finally, the Pathfinder algorithm is applied for corresponding path planning. The framework is implemented in the Artificial Intelligence (AI) Habitat simulator, and the case studies demonstrate that the AGV can accurately comprehend complex language instructions, empowering human operators to complete manufacturing tasks efficiently.

Islamic QA with Chatbot System Using Convolutional Neural Network

Many questions and answers about Islamic law are scattered on the internet and have been explained repeatedly by various sites. One solution is presented by the website www.piss-ktb.com, which creates a web-based source of information in the form of Frequently Asked Questions (FAQ). However, web-based FAQs have a weakness: users still have to browse through the available questions one by one according to the questions they want to know the answers to. Browsing through thousands of FAQs is inefficient and exhausting. Thus, a chatbot system can become a better alternative to the FAQ website. Still, chatbots are difficult to use because most of their conversations are hard to understand. A single character error will cause the system to misunderstand its meaning. In reality, users expect a chatbot that can understand everyday language. Thus, it is necessary to develop a chatbot system that can understand various common sentence combinations in everyday language and understand the meaning of words. In addition, it should be able to predict answers automatically to various kinds of questions and requests, even though the initial training data is relatively low. Therefore, this study aims to develop a system that can provide answers automatically based on user commands in natural language using Global Vectors for Word Representations (GloVe), Convolutional Neural Networks (CNN), and Transfer Learning techniques. The result shows that the use of transfer learning and the Nadam optimizer can improve the system’s performance.

Natural language command parsing for agricultural measurement and control based on AMR and entity recognition

There is scope to enhance agricultural measurement and control systems user interactivity, which typically necessitates training for users to perform specific operations successfully. With the continuous development of natural language semantic processing technology, it has become essential to augment the user-friendliness of multifaceted control and query operations in the agricultural measurement and control sector, ultimately leading to reduced operation costs for users. The study aims to focus on command parsing. The proposed AMR-OPO semantic parsing framework is based on the natural language understanding method of Abstract Meaning Representation of Rooted Markup Graphs (AMR). It transforms the user’s natural language inputs into structured ternary (OPO) statements (operation-place-object) and converts the corresponding parameters of the user’s input commands. The framework subsequently sends the transformed commands to the relevant devices via the IoT gateway. To tackle the intricate task of parsing instructions, we developed a BERT-BiLSTM-ATT-CRF-OPO entity recognition model. This model can detect and extract entities from agricultural instructions, and precisely populate them into OPO statements. Our model shows exceptional accuracy in instruction parsing, with precision, recall, and F-value all measuring at 92.13%, 93.12%, and 92.76%, correspondingly. The findings from our experiment reveal outstanding and precise performance of our approach. It is anticipated that our algorithm will enhance the user experience offered by agricultural measurement and control systems, while also making them more user-friendly.

Design and Development of Automated Control of Restaurant Robot

This paper presents a novel voice-controlled restaurant robot system designed to revolutionize the dining experience by seamlessly integrating advanced robotics and natural language processing technologies. The proposed system aims to enhance customer service, operational efficiency, and overall dining satisfaction by providing a user-friendly and interactive platform. .The restaurant robot is equipped with a comprehensive set of capabilities, including, voice recognition and manipulation. By leveraging voice control, users can interact with the robot using natural language commands, enabling seamless communication and intuitive control. This allows customers to effortlessly place orders The system's is comprises a mobile robotic platform, bluetooth modules, a cloud-based natural language processing, The voice control functionality is implemented through a sophisticated speech recognition system, enabling the robot to accurately understand and interpret user commands. .In addition to customer-facing interactions, the restaurant robot also assists restaurant staff in various tasks. Deliver food and beverages to tables, handle payment transactions, and perform routine cleaning tasks. By automating these processes, the system improves operational efficiency, reduces labor costs, and allows human staff to focus on highervalue activities such as customer engagement and culinary expertise. .To evaluate the performance and usability of the voice-controlled restaurant robot, a series of experiments were conducted in a real-world restaurant environment. The results demonstrated the system's high recognition accuracy, robustness in noisy environments, and its ability to effectively handle a wide range of user requests. .The proposed voice-controlled restaurant robot presents a promising solution for transforming the dining experience. It combines the convenience of voice control with advanced robotics, enabling seamless and interactive interactions between customers, staff, and the robotic platform. The system's ability to enhance customer service, operational efficiency, and overall dining satisfaction positions it as a significant advancement in the hospitality industry.

Towards a Natural Language Interface for Flexible Multi-Agent Task Assignment

Task assignment and scheduling algorithms are powerful tools for autonomously coordinating large teams of robotic or AI agents. However, the decisions these system make often rely on components designed by domain experts, which can be difficult for non-technical end-users to understand or modify to their own ends. In this paper we propose a preliminary design for a flexible natural language interface for a task assignment system. The goal of our approach is both to grant users more control over a task assignment system's decision process, as well as render these decisions more transparent. Users can direct the task assignment system via natural language commands, which are applied as constraints to a mixed-integer linear program (MILP) using a large language model (LLM). Additionally, our proposed system can alert users to potential issues with their commands, and engage them in a corrective dialogue in order to find a viable solution. We conclude with a description of our planned user-evaluation in the simulated environment Overcooked and describe next steps towards developing a flexible and transparent task allocation system.

An Investigation on Utilizing Large Language Model for Industrial Computer-Aided Design Automation

Text-to-image or 3D content generation has attracted attention in academia and industry because of its impressive performance in high-fidelity content generation with natural language commands. However, its widespread application in industrial computation-aided design (CAD) is not yet found. The main issue lies in the end-to-end generation through implicit probabilistic learning from datasets. This presents difficulties in precise, controllable, and transparent creations. Also, it is time-consuming and labor-intensive for designers to verify and customize the generated result to fit their demands. Furthermore, popular generative models offer limited support for early-stage engineering computations, leaving tedious and massive calculations to human engineers. To overcome these challenges, this paper proposes a method to apply large language model (LLM) to assist standard CAD design workflow on both engineering computation and precise 3D geometry generation. During the designing process, LLM can support the computation process with human guidance and generate accurate 3D shapes with the capability of sequential code representation and transformation. To verify the feasibility of the proposed method, a use case for a one-stage reduction gear system consisting of two gears will be given as a proof-of-concept demonstration. The result shows that the LLM can carry out the computation and get the result as expected with only a few lines of human guidance and feedback. Then, it can create CAD models with accurate parameters aligned to the computational results. It can cope with various requirements given by users and allow users to further modify the generation using the same CAD environment without much effort. Authors believe that this piece of work can shed light on utilizing the capability of LLMs for a certain part of task automation based on human beings’ intentions. With the rise of LLMs and relevant technologies, conventional CAD may shift to a more user-friendly AI-aided design (AIAD) diagram.

Data Playwright: Authoring Data Videos With Annotated Narration.

Creating data videos that effectively narrate stories with animated visuals requires substantial effort and expertise. A promising research trend is leveraging the easy-to-use natural language (NL) interaction to automatically synthesize data video components from narrative content like text narrations, or NL commands that specify user-required designs. Nevertheless, previous research has overlooked the integration of narrative content and specific design authoring commands, leading to generated results that lack customization or fail to seamlessly fit into the narrative context. To address these issues, we introduce a novel paradigm for creating data videos, which seamlessly integrates users' authoring and narrative intents in a unified format called annotated narration, allowing users to incorporate NL commands for design authoring as inline annotations within the narration text. Informed by a formative study on users' preference for annotated narration, we develop a prototype system named Data Playwright that embodies this paradigm for effective creation of data videos. Within Data Playwright, users can write annotated narration based on uploaded visualizations. The system's interpreter automatically understands users' inputs and synthesizes data videos with narration-animation interplay, powered by large language models. Finally, users can preview and fine-tune the video. A user study demonstrated that participants can effectively create data videos with Data Playwright by effortlessly articulating their desired outcomes through annotated narration.

Exploring the role of artificial intelligence in the human era involves a multidimensional perspective that encompasses technological, ethical, social, and economic considerations

The artificial intelligence (AI) device is powered by an advanced Snapdragon platform from Qualcomm Technologies and boasts a range of sensors that enable contextual and ambient compute interactions. The Humane AI Pin offers a plethora of features that enhance our daily lives, including Contextual AI: The device can understand and respond to our natural language commands, allowing us to control music, make calls, send texts, and set reminders with ease. Ambient Awareness: The AI Pin's sensors provide real-time information about our surroundings, enabling it to adjust audio volume, provide notifications, and translate languages in context. Seamless Integration: The device eliminates the need for a smartphone, allowing us to interact with AI without the need to constantly look at a screen. Establishing robust frameworks for AI governance, promoting transparency, and prioritizing ethical considerations will be essential in harnessing the full potential of AI while minimizing its risks. Ultimately, a multidimensional perspective is crucial to charting a course for AI that aligns with human values, fosters responsible innovation, and enhances the overall well-being of society in the human era.

Speech Recognition and Sending Alert Message If Abusive Words are Detected

Abstract: This paper introduces an innovative approach to bolstering the field of speech recognition. Speech recognition, a pivotal process in facilitating computer systems' understanding and response to natural language commands, has garnered considerable importance in today's digital landscape. Our novel method aims to enable users to issue voice commands with enhanced accuracy and efficiency. This approach encompasses several critical steps: the intake of a single audio input, effective noise reduction, and the utilization of pattern matching for audio recognition. To fortify the system's robustness, we've integrated an error tolerance mechanism capable of accommodating human errors and mitigating environmental noise.What sets our method apart is its ability to circumvent the need for storing extensive audio data for matching, leading to a substantial reduction in both space and time complexity when compared to conventional systems. In our experiments, we have exclusively employed a database containing an array of predefined voice patterns, effectively eliminating the necessity for end-user training.

Large language models for chemistry robotics

This paper proposes an approach to automate chemistry experiments using robots by translating natural language instructions into robot-executable plans, using large language models together with task and motion planning. Adding natural language interfaces to autonomous chemistry experiment systems lowers the barrier to using complicated robotics systems and increases utility for non-expert users, but translating natural language experiment descriptions from users into low-level robotics languages is nontrivial. Furthermore, while recent advances have used large language models to generate task plans, reliably executing those plans in the real world by an embodied agent remains challenging. To enable autonomous chemistry experiments and alleviate the workload of chemists, robots must interpret natural language commands, perceive the workspace, autonomously plan multi-step actions and motions, consider safety precautions, and interact with various laboratory equipment. Our approach, CLAIRify, combines automatic iterative prompting with program verification to ensure syntactically valid programs in a data-scarce domain-specific language that incorporates environmental constraints. The generated plan is executed through solving a constrained task and motion planning problem using PDDLStream solvers to prevent spillages of liquids as well as collisions in chemistry labs. We demonstrate the effectiveness of our approach in planning chemistry experiments, with plans successfully executed on a real robot using a repertoire of robot skills and lab tools. Specifically, we showcase the utility of our framework in pouring skills for various materials and two fundamental chemical experiments for materials synthesis: solubility and recrystallization. Further details about CLAIRify can be found at https://ac-rad.github.io/clairify/.

Converting Natural Language To SQL Queries Using LSTM

bstract: In today's world, not everyone is familiar with using Structured Query Language (SQL). This makes it hard for users to understand or create complex SQL queries. What we need is an improved application with a smarter interface that can bridge the gap between novice users and databases. Databases are great at managing data, but to understand their structure, users have to learn SQL. This poses a challenge for non-experts who aren't well-versed in SQL. What they need is a system that allows them to interact with databases in natural language. The system should be capable of understanding and responding to natural language commands. To achieve this objective, we utilize a range of end-to-end deep learning models, as well as probability models like conditional random field. The ambiguity in natural language makes it exceedingly difficult to determine the exact meaning of every word, therefore it is a difficult process to map individual keywords to the description of the schema and the contents of the underlying database. Accurate predictions can help us avoid unnecessary trouble. If we apply machine learning tools, we can skip the complicated process.

A Deep Learning-Based Chinese Semantic Parser for the Almond Virtual Assistant.

Almond is an extendible open-source virtual assistant designed to help people access Internet services and IoT (Internet of Things) devices. Both are referred to as skills here. Service providers can easily enable their devices for Almond by defining proper APIs (Application Programming Interfaces) for ThingTalk in Thingpedia. ThingTalk is a virtual assistant programming language, and Thingpedia is an application encyclopedia. Almond uses a large neural network to translate user commands in natural language into ThingTalk programs. To obtain enough data for the training of the neural network, Genie was developed to synthesize pairs of user commands and corresponding ThingTalk programs based on a natural language template approach. In this work, we extended Genie to support Chinese. For 107 devices and 261 functions registered in Thingpedia, 649 Chinese primitive templates and 292 Chinese construct templates were analyzed and developed. Two models, seq2seq (sequence-to-sequence) and MQAN (multiple question answer network), were trained to translate user commands in Chinese into ThingTalk programs. Both models were evaluated, and the experiment results showed that MQAN outperformed seq2seq. The exact match, BLEU, and F1 token accuracy of MQAN were 0.7, 0.82, and 0.88, respectively. As a result, users could use Chinese in Almond to access Internet services and IoT devices registered in Thingpedia.

Building a Natural Language Query and Control Interface for IoT Platforms

The rapid development and popularity of IoT technology has reshaped the way people interact with the real world. Many researchers have attempted to build natural language interfaces for IoT platforms, but have not produced much progress in parsing natural language commands that contain multiple operations and more complex logical structures. In this paper, we propose IoT-NLI, a natural language query and control interface for popular IoT platforms, which uses hierarchical semantic parsing algorithms and directed edge-tagged graph structures to efficiently parse natural language commands input by users, enabling them to perform multiple operations contained in one complex natural language command. Experiments in three domains, agriculture, industry, and smart home, show that IoT-NLI has excellent performance and reasonable response time. Finally, a IoT-NLI application was developed on the Android platform and integrated with the AliCloud platform. It enables users to query and control devices on Android phones through chat windows similar to instant messaging software.

Talk2Car: Predicting Physical Trajectories for Natural Language Commands

In recent years, there has been an increased interest in giving verbal commands to self-driving cars. Even though multiple companies have showcased progress towards fully autonomous vehicles, surveys have indicated that people are wary of relinquishing total control of the vehicle to the AI. Thus, a system allowing passengers to control the vehicle’s actions would be preferable. Natural language, the most widespread form of communication among humans, presents itself as the most natural control interface, and survey results confirm that the ability to give verbal commands to self-driving vehicles would make the passengers more at ease. In this work, we propose a novel system that predicts which object is referred to by the issued command and the path the car should follow through the immediate surroundings to execute the command. We experiment with different approaches and features to predict the object of interest and show that our simple but effective approach achieves state-of-the-art performance. For predicting the trajectory, we propose a model that relies on a mixture density approach for modeling the distributions of key waypoints of the trajectory in the top-down scene layout. Additionally, we investigate the influence of the two tasks on each other and show that improvements in the prediction of the object of interest lead to improvements in the trajectory prediction task. Finally, we provide the research community with an extension to the Talk2Car dataset, with new trajectory annotations for given commands.

Conversational Neuro-Symbolic Commonsense Reasoning

In order for conversational AI systems to hold more natural and broad-ranging conversations, they will require much more commonsense, including the ability to identify unstated presumptions of their conversational partners. For example, in the command "If it snows at night then wake me up early because I don't want to be late for work" the speaker relies on commonsense reasoning of the listener to infer the implicit presumption that they wish to be woken only if it snows enough to cause traffic slowdowns. We consider here the problem of understanding such imprecisely stated natural language commands given in the form of if-(state), then-(action), because-(goal) statements. More precisely, we consider the problem of identifying the unstated presumptions of the speaker that allow the requested action to achieve the desired goal from the given state (perhaps elaborated by making the implicit presumptions explicit). We release a benchmark data set for this task, collected from humans and annotated with commonsense presumptions. We present a neuro-symbolic theorem prover that extracts multi-hop reasoning chains, and apply it to this problem. Furthermore, to accommodate the reality that current AI commonsense systems lack full coverage, we also present an interactive conversational framework built on our neuro-symbolic system, that conversationally evokes commonsense knowledge from humans to complete its reasoning chains.

Ontology-based knowledge management with verbal interaction for command interpretation and execution by home service robots

This paper describes a system for service robots that combines ontological knowledge reasoning and human–robot interaction to interpret natural language commands and successfully perform household chores, such as finding and delivering objects. Knowledge and context reasoning is essential for providing more efficient service robots, given their diverse and continuously changing environments. Moreover, since they are in contact with humans, robots require such skills as interaction and language. Therefore, we developed a system with specific modules to manage robots’ knowledge and reasoning, command analysis, decision-making, and talking interaction. The system relies on inference methods and verbal interaction to understand commands and clarify uncertain information. We tested our system inside a simulated environment where the robot receives commands with missing or unclear information. The system’s performance was compared with the average performance of human subjects who completed the same commands in the simulation.

AI based Computational Trust Model for Intelligent Virtual Assistant

The Intelligent virtual assistant (IVA) also called AI assistant or digital assistant is software developed as a product by organizations like Google, Apple, Microsoft and Amazon. Virtual assistant based on Artificial Intelligence which works and processes on natural language commands given by humans. It helps the user to work more efficiently and also saves time. It is human friendly as it works on natural language commands given by humans. Voice-controlled Intelligent Virtual Assistants (IVAs) have seen gigantic development as of late on cell phones and as independent gadgets in individuals’ homes. The intelligent virtual assistant is very useful for illiterate and visually impaired people around the world. While research has analyzed the expected advantages and downsides of these gadgets for IVA clients, barely any investigations have exactly assessed the need of security and trust as a singular choice to use IVAs. In this proposed work, different IPA users and non-users (N=1000) are surveyed to understand and analyze the barriers and motivations to adopting IPAs and how users are concerned about data privacy and trust with respect to organizational compliances and social contract related to IPA data and how these concerns have affected the acceptance and use of IPAs. We have used Naïve Byes Classifier to compute trust in IVA devices and further evaluate probability of using different trusted IVA devices.

Natural mapping between voice commands and APIs

Abstract After a voice control system transforms audio input into a natural language sentence, its main purpose is to map this sentence to a specific action in the API (application programming interface) that should be performed. This mapping is usually specified after the API is already designed. In this paper, we show how an API can be designed with voice control in mind, which makes this mapping natural. The classes, methods, and parameters in the source code are named and typed according to the terms expected in the natural language commands. When this is insufficient, annotations (attribute-oriented programming) are used to define synonyms, string-to-object maps, or other properties. We also describe the mapping process and present a preliminary implementation called VCMapper. In its evaluation on a third-party dataset, it was successfully used to map all the sentences, while a large portion of the mapping was performed using only naming and typing conventions.