Interactive Command Research Articles

ChatIoT: Zero-code Generation of Trigger-action Based IoT Programs

Trigger-Action Program (TAP) is a simple but powerful format to realize intelligent IoT applications, especially in home automation scenarios. Existing trace-driven approaches and in-situ programming approaches depend on either customized interaction commands or well-labeled datasets, resulting in limited applicable scenarios. In this paper, we propose ChatIoT, a zero-code TAP generation system based on large language models (LLMs). With a novel context-aware compressive prompting scheme, ChatIoT is able to automatically generate TAPs from user requests in a token-efficient manner and deploy them to the TAP runtime. Further, for those TAP requests including unknown sensing abilities, ChatIoT can also generate new AI models with knowledge distillation by multimodal LLMs, with a novel model customization method based on deep reinforcement learning. We implemented ChatIoT and evaluated its performance extensively. Results show that ChatIoT can reduce token consumption by 26.1-84.9% and improve TAP generation accuracy by 4.2-65.5% compared to state-of-the-art approaches in multiple settings. We also conducted a real user study, and ChatIoT can achieve 91.57% TAP generation accuracy.

Lipwatch: Enabling Silent Speech Recognition on Smartwatches using Acoustic Sensing

Silent Speech Interfaces (SSI) on mobile devices offer a privacy-friendly alternative to conventional voice input methods. Previous research has primarily focused on smartphones. In this paper, we introduce Lipwatch, a novel system that utilizes acoustic sensing techniques to enable SSI on smartwatches. Lipwatch leverages the inaudible waves emitted by the watch's speaker to capture lip movements and then analyzes the echo to enable SSI. In contrast to acoustic sensing-based SSI on smartphones, our development of Lipwatch takes into full consideration the specific scenarios and requirements associated with smartwatches. Firstly, we elaborate a wake-up-free mechanism, allowing users to interact without the need for a wake-up phrase or button presses. The mechanism utilizes the inertial sensors on the smartwatch to detect gestures, in combination with acoustic signals that detecting lip movements to determine whether SSI should be activated. Secondly, we design a flexible silent speech recognition mechanism that explores limited vocabulary recognition to comprehend a broader range of user commands, even those not present in the training dataset, relieving users from strict adherence to predefined commands. We evaluate Lipwatch on 15 individuals using a set of the 80 most common interaction commands on smartwatches. The system achieves a Word Error Rate (WER) of 13.7% in user-independent test. Even when users utter commands containing words absent in the training set, Lipwatch still demonstrates a remarkable 88.7% top-3 accuracy. We implement a real-time version of Lipwatch on a commercial smartwatch. The user study shows that Lipwatch can be a practical and promising option to enable SSI on smartwatches.

Simulation-based Optimization of User Interfaces for Quality-assuring Machine Learning Model Predictions

Quality-sensitive applications of machine learning (ML) require quality assurance (QA) by humans before the predictions of an ML model can be deployed. QA for ML (QA4ML) interfaces require users to view a large amount of data and perform many interactions to correct errors made by the ML model. An optimized user interface (UI) can significantly reduce interaction costs. While UI optimization can be informed by user studies evaluating design options, this approach is not scalable, because there are typically numerous small variations that can affect the efficiency of a QA4ML interface. Hence, we propose using simulation to evaluate and aid the optimization of QA4ML interfaces. In particular, we focus on simulating the combined effects of human intelligence in initiating appropriate interaction commands and machine intelligence in providing algorithmic assistance for accelerating QA4ML processes. As QA4ML is usually labor-intensive, we use the simulated task completion time as the metric for UI optimization under different interface and algorithm setups. We demonstrate the usage of this UI design method in several QA4ML applications.

An open-source human-in-the-loop BCI research framework: method and design.

Brain-computer interfaces (BCIs) translate brain activity into digital commands for interaction with the physical world. The technology has great potential in several applied areas, ranging from medical applications to entertainment industry, and creates new conditions for basic research in cognitive neuroscience. The BCIs of today, however, offer only crude online classification of the user's current state of mind, and more sophisticated decoding of mental states depends on time-consuming offline data analysis. The present paper addresses this limitation directly by leveraging a set of improvements to the analytical pipeline to pave the way for the next generation of online BCIs. Specifically, we introduce an open-source research framework that features a modular and customizable hardware-independent design. This framework facilitates human-in-the-loop (HIL) model training and retraining, real-time stimulus control, and enables transfer learning and cloud computing for the online classification of electroencephalography (EEG) data. Stimuli for the subject and diagnostics for the researcher are shown on separate displays using web browser technologies. Messages are sent using the Lab Streaming Layer standard and websockets. Real-time signal processing and classification, as well as training of machine learning models, is facilitated by the open-source Python package Timeflux. The framework runs on Linux, MacOS, and Windows. While online analysis is the main target of the BCI-HIL framework, offline analysis of the EEG data can be performed with Python, MATLAB, and Julia through packages like MNE, EEGLAB, or FieldTrip. The paper describes and discusses desirable properties of a human-in-the-loop BCI research platform. The BCI-HIL framework is released under MIT license with examples at: bci.lu.se/bci-hil (or at: github.com/bci-hil/bci-hil).

Numerical studies on the behavior of cold‐formed double‐channel shear connector

AbstractThe longitudinal shear forces are transferred through mechanical shear connectors at the steel beams and concrete slab intersection in composite beams. The shear connectors can be directly welded to the hot rolled steel beam, which is difficult in thin, cold‐formed steel beams. Thus, a cold‐formed double‐channel shear connector is proposed, and its suitability is investigated in this study. The double channel shear connector is a channel shaped shear connector formed from a flat plat and it is mounted on the flange of steel beam in pairs. The finite element model of the experimental Pushout test assembly was developed with the equivalent numerical constraints and interaction commands in a finite element software (ABAQUS). The developed models were fed with initial imperfection, load slip response, and material nonlinearity to achieve the proper experiment simulation. Initially, the developed models were validated against nine experimental test data reported by authors in the previous study. The parametric study was conducted to assess the behavior of cold‐formed channel shear connectors. The key parameters adopted in this study are the length of the shear connector, the grade of concrete and the grade and thickness of the steel connector. Three common failure patterns were witnessed among 107 numerical models. In this study, the factors influencing the control of failure patterns are presented, along with the factors influencing the shear connector's ultimate capacity.

Exploring Interactions with Printed Data Visualizations in Augmented Reality.

This paper presents a design space of interaction techniques to engage with visualizations that are printed on paper and augmented through Augmented Reality. Paper sheets are widely used to deploy visualizations and provide a rich set of tangible affordances for interactions, such as touch, folding, tilting, or stacking. At the same time, augmented reality can dynamically update visualization content to provide commands such as pan, zoom, filter, or detail on demand. This paper is the first to provide a structured approach to mapping possible actions with the paper to interaction commands. This design space and the findings of a controlled user study have implications for future designs of augmented reality systems involving paper sheets and visualizations. Through workshops ( N=20) and ideation, we identified 81 interactions that we classify in three dimensions: 1) commands that can be supported by an interaction, 2) the specific parameters provided by an (inter)action with paper, and 3) the number of paper sheets involved in an interaction. We tested user preference and viability of 11 of these interactions with a prototype implementation in a controlled study ( N=12, HoloLens 2) and found that most of the interactions are intuitive and engaging to use. We summarized interactions (e.g., tilt to pan) that have strong affordance to complement "point" for data exploration, physical limitations and properties of paper as a medium, cases requiring redundancy and shortcuts, and other implications for design.

Towards the Design and Implementation of a Programming Language (Beex)

Software Engineers, Computer Scientists, and Software Experts alike are faced to decide which programming language is best suited for a certain purpose as the use of programming languages grows. When we consider the various types of programming languages available today, such as Domain Specific Languages (DSL), General Purpose Languages (GPL), Functional Programming Languages (FPL), Imperative Programming Languages (IPL), amongst others, this becomes complicated. In this study, we introduce BeeX, an interpreted language, with the aim of showing the process and principles involved in language design and consider various choices faced by language designers of various programming languages. BeeX was created with simplicity in mind, thus the study focused on architectural design options. We look at the implementation standpoint and try to figure out what the basic building parts of most programming languages are, such as lexical analysis, syntax analysis, and evaluation phase. To achieve this, we created an interactive command interface that evaluated various BeeX language constructs(conditional logic statements, arithmetic expressions, loop constructs etc.) which allowed students to easily experiment with the proposed language. The results of the tests showed that students and programmers alike can use the BeeX programming language to create a variety of code structures that are simple to use.

EOG-Based Human-Computer Interface: 2000-2020 Review.

Electro-oculography (EOG)-based brain–computer interface (BCI) is a relevant technology influencing physical medicine, daily life, gaming and even the aeronautics field. EOG-based BCI systems record activity related to users’ intention, perception and motor decisions. It converts the bio-physiological signals into commands for external hardware, and it executes the operation expected by the user through the output device. EOG signal is used for identifying and classifying eye movements through active or passive interaction. Both types of interaction have the potential for controlling the output device by performing the user’s communication with the environment. In the aeronautical field, investigations of EOG-BCI systems are being explored as a relevant tool to replace the manual command and as a communicative tool dedicated to accelerating the user’s intention. This paper reviews the last two decades of EOG-based BCI studies and provides a structured design space with a large set of representative papers. Our purpose is to introduce the existing BCI systems based on EOG signals and to inspire the design of new ones. First, we highlight the basic components of EOG-based BCI studies, including EOG signal acquisition, EOG device particularity, extracted features, translation algorithms, and interaction commands. Second, we provide an overview of EOG-based BCI applications in the real and virtual environment along with the aeronautical application. We conclude with a discussion of the actual limits of EOG devices regarding existing systems. Finally, we provide suggestions to gain insight for future design inquiries.

An empirical investigation of command-line customization

The interactive command line, also known as the shell, is a prominent mechanism used extensively by a wide range of software professionals (engineers, system administrators, data scientists, etc.). Shell customizations can therefore provide insight into the tasks they repeatedly perform, how well the standard environment supports those tasks, and ways in which the environment could be productively extended or modified. To characterize the patterns and complexities of command-line customization, we mined the collective knowledge of command-line users by analyzing more than 2.2 million shell alias definitions found on GitHub. Shell aliases allow command-line users to customize their environment by defining arbitrarily complex command substitutions. Using inductive coding methods, we found three types of aliases that each enable a number of customization practices: Shortcuts (for nicknaming commands, abbreviating subcommands, and bookmarking locations), Modifications (for substituting commands, overriding defaults, colorizing output, and elevating privilege), and Scripts (for transforming data and chaining subcommands). We conjecture that identifying common customization practices can point to particular usability issues within command-line programs, and that a deeper understanding of these practices can support researchers and tool developers in designing better user experiences. In addition to our analysis, we provide an extensive reproducibility package in the form of a curated dataset together with well-documented computational notebooks enabling further knowledge discovery and a basis for learning approaches to improve command-line workflows.

Mapping Directional Mid-Air Unistroke Gestures to Interaction Commands: A User Elicitation and Evaluation Study

A stroke is the basic limb movement that both humans and animals naturally and repetitiously perform. Having been introduced into gestural interaction, mid-air stroke gestures saw a wide application range and quite intuitive use. In this paper, we present an approach for building command-to-gesture mapping that exploits the semantic association between interactive commands and the directions of mid-air unistroke gestures. Directional unistroke gestures make use of the symmetry of the semantics of commands, which makes a more systematic gesture set for users’ cognition and reduces the number of gestures users need to learn. However, the learnability of the directional unistroke gestures is varying with different commands. Through a user elicitation study, a gesture set containing eight directional mid-air unistroke gestures was selected by subjective ratings of the direction in respect to its association degree with the corresponding command. We evaluated this gesture set in a following study to investigate the learnability issue, and the directional mid-air unistroke gestures and user-preferred freehand gestures were compared. Our findings can offer preliminary evidence that “return”, “save”, “turn-off” and “mute” are the interaction commands more applicable to using directional mid-air unistrokes, which may have implication for the design of mid-air gestures in human–computer interaction.

A multistudy investigation of drivers and passengers’ gesture and voice input preferences for in-vehicle interactions

We conduct an examination of the preferences of drivers and passengers alike for in-vehicle interactions with a multistudy approach consisting of (1) a targeted literature survey of applications and user interfaces designed to support interactions with in-vehicle controls and systems based on gesture and voice input; (2) a large-scale survey (N = 160 participants) to understand drivers and passengers’ preferences for driving and traveling by car; and (3) an end-user elicitation study (N = 40 drivers and passengers) to collect preferences for gesture and voice input commands for in-vehicle interaction. We analyze and discuss the gesture and voice commands proposed by our participants and describe their characteristics, such as production times for gesture input and the vocabulary size of voice commands.

Detecting malware communities using socio-cultural cognitive mapping

We apply a variation of socio-cultural cognitive mapping (SCM) to computer malware features explored previously by Saxe and Berlin that characterized malware binaries as benign or malicious based on 1024 program features derived from a deep neural network-based detection system. In this work, we model the features as attributes within a latent spatial domain using a weighted consensus graph representation to visualize and analyze the malware binary communities. The data used in our analysis is extracted from a Remote Access Trojan family named Sakula that first appeared in 2012, and has been used to enable an adversary to run interactive commands and execute remote program functions. Our results show that by SCM we were able to identify distinct malware communities within the malware family, which revealed insights into the overall structure of the various binaries as well as possible temporal relationships between the binaries.

Using the Object-Oriented PowerShell for Simple Proteomics Data Analysis.

Scripting languages such as Python and Bash are appreciated for solving simple, everyday tasks in bioinformatics. A more recent, object-oriented command shell and scripting language, PowerShell, has many attractive features: an object-oriented interactive command line, fluent navigation and manipulation of XML files, ability to explore and consume Web services from the command line, consistent syntax and grammar, rich regular expressions, and advanced output formatting. The key difference between classical command shells and scripting languages, such as bash, and object-oriented ones, such as PowerShell, is that in the latter the result of a command is a structured object with inherited properties and methods rather than a simple stream of characters. Conveniently, PowerShell is included in all new releases of Microsoft Windows and is available for Linux and macOS, making any data processing script portable. In this chapter we demonstrate how PowerShell in particular allows easy interaction with mass spectrometry data in XML formats, connection to Web services for tools such as BLAST, and presentation of results as formatted text or graphics. These features make PowerShell much more than "yet another scripting language."

Multiscale finite element calculations in Python using SfePy

SfePy (Simple finite elements in Python) is a software for solving various kinds of problems described by partial differential equations in one, two or three spatial dimensions by the finite element method. Its source code is mostly (85\%) Python and relies on fast vectorized operations provided by the NumPy package. For a particular problem two interfaces can be used: a declarative application programming interface (API), where problem description/definition files (Python modules) are used to define a calculation, and an imperative API, that can be used for interactive commands, or in scripts and libraries. After outlining the SfePy package development, the paper introduces its implementation, structure and general features. The components for defining a partial differential equation are described using an example of a simple heat conduction problem. Specifically, the declarative API of SfePy is presented in the example. To illustrate one of SfePy's main assets, the framework for implementing complex multiscale models based on the theory of homogenization, an example of a two-scale piezoelastic model is presented, showing both the mathematical description of the problem and the corresponding code.

BioKEEN: a library for learning and evaluating biological knowledge graph embeddings.

Knowledge graph embeddings (KGEs) have received significant attention in other domains due to their ability to predict links and create dense representations for graphs' nodes and edges. However, the software ecosystem for their application to bioinformatics remains limited and inaccessible for users without expertise in programing and machine learning. Therefore, we developed BioKEEN (Biological KnowlEdge EmbeddiNgs) and PyKEEN (Python KnowlEdge EmbeddiNgs) to facilitate their easy use through an interactive command line interface. Finally, we present a case study in which we used a novel biological pathway mapping resource to predict links that represent pathway crosstalks and hierarchies. BioKEEN and PyKEEN are open source Python packages publicly available under the MIT License at https://github.com/SmartDataAnalytics/BioKEEN and https://github.com/SmartDataAnalytics/PyKEEN. Supplementary data are available at Bioinformatics online.

Verbal and vibrotactile cues on multiuser usability within collaborative virtual environments

Suitable cues for communication are crucial to multiuser usability of collaborative virtual environments (VEs). Although verbal cues are common among users, vibrotactile cues are nonverbal to facilitate conventionally interaction with objects. Few research efforts have examined the role of verbal and vibrotactile cues on multiuser usability of a collaborative VE. This paper hence presents assessments of verbal and vibrotactile cues, as well their combination, on affecting multiuser usability of the VE. The VE resolved conflicts of interactive commands issued by peer users through a dynamic priority (DP) model, which granted interaction to a user based on the recency of his/her gained interactions. As revealed in the existing work, the DP model yielded perceived equality in interaction (EII) among users to promote multiuser satisfaction. The assessments utilized metrics of multiuser usability, which we proposed for the components of both ISO/IEC 2050:2011 and 25022:2016 standards. The metrics considered collective outcomes of the users, differing from existing metrics of measuring individual experiences. We first conducted two baseline evaluations within the VE. One evaluation investigated how verbal cues affect EII and multiuser usability; another evaluation studied how vibrotactile cues influence the effect of the DP model. Based on the evaluations, we then undertook a study to assess the effect of combining verbal and vibrotactile cues on multiuser usability of the VE. We observed that both cues had no effect on the DP model to yield perceived EII. However, the combination of verbal cues in an affirmative mode and vibrotactile cues in a counteractive mode enhanced significantly the multiuser usability of the VE for the task of the study, compared to individual verbal and vibrotactile cues. These observations indicate a suitable combination of both cues to potentially foster multiuser usability of collaborative VEs.

A Novel Gesture Recognition System for Intelligent Interaction with a Nursing-Care Assistant Robot

The expansion of nursing-care assistant robots in smart infrastructure has provided more applications for homecare services, which has raised new demands for smart and natural interaction between humans and robots. This article proposed an innovative hand motion trajectory (HMT) gesture recognition system based on background velocity features. Here, a new wearable wrist-worn camera prototype for gesture’s video collection was designed, and a new method for the segmentation of continuous gestures was shown. Meanwhile, a nursing-care assistant robot prototype was designed for assisting the elderly, which is capable of carrying the elderly with omnidirectional motion and grabbing the specified object at home. In order to evaluate the performance of the gesture recognition system, 10 special gestures were defined as the move commands for interaction with the robot, and 1000 HMT gesture samples were obtained from five subjects for leave-one-subject-out (LOSO) cross-validation classification with an average recognition accuracy of up to 97.34%. Moreover, the performance and practicability of the proposed system were further demonstrated by controlling the omnidirectional movement of the nursing-care assistant robot using the predefined gesture commands.

Spatial landscape model to characterize biological diversity using R statistical computing environment

Due to urbanization and population growth, the degradation of natural forests and associated biodiversity are now widely recognized as a global environmental concern. Hence, there is an urgent need for rapid assessment and monitoring of biodiversity on priority using state-of-art tools and technologies. The main purpose of this research article is to develop and implement a new methodological approach to characterize biological diversity using spatial model developed during the study viz. Spatial Biodiversity Model (SBM). The developed model is scale, resolution and location independent solution for spatial biodiversity richness modelling. The platform-independent computation model is based on parallel computation. The biodiversity model based on open-source software has been implemented on R statistical computing platform. It provides information on high disturbance and high biological richness areas through different landscape indices and site specific information (e.g. forest fragmentation (FR), disturbance index (DI) etc.). The model has been developed based on the case study of Indian landscape; however it can be implemented in any part of the world. As a case study, SBM has been tested for Uttarakhand state in India. Inputs for landscape ecology are derived through multi-criteria decision making (MCDM) techniques in an interactive command line environment. MCDM with sensitivity analysis in spatial domain has been carried out to illustrate the model stability and robustness. Furthermore, spatial regression analysis has been made for the validation of the output.

Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources

In the area of Human-Computer Interaction (HCI), the concept of Around Device Interaction (ADI) is gaining special attention nowadays. It provides the possibility of interacting with the device by making use of gestures made in the surrounding space, as well as by detecting certain ambient changes, thus eliminating the need of direct physical contact. Contact-free interaction can be useful for wearable and smart devices, especially in cases where operating with small buttons and touchscreens appears to be particularly inconvenient. This paper presents alternative techniques for smartphone interaction based on different sensory inputs. Namely, external sources of magnetic field, light, and infrared (IR) signals are used to manipulate device sensors' readings which are in turn interpreted as the corresponding interaction commands. Proof-of-concept mobile applications are developed and here presented, demonstrating both the potentials and utility of the contact-free interaction. Specifically, small neodymium magnet was used for smartphone text entry, typical battery flashlight served for sending information to device via Morse code, as well as for alarm triggering, and voice-enabled control device was utilized for invoking smartphone calls from a distance. The presented interaction techniques do not require smartphone hardware alterations. While the usage of built-in magnetic field and ambient light sensors is assumed, peripheral infrared receiver can be used in order to provide IR-based support.

Towards Racing Gamification with Natural Interface for post stroke Rehabilitation

Stroke patient basically suffer from limited movement, they cannot control their balance very well, therefore the therapy that involve repetitive motion, harmonization and stability workout should be applied to them. The invention of Kinect has lead people to applied this device as a tool for recovering patient from stroke because its capability on tracking the skeletal of human body. Racing Game is very popular among the adults, elderly and even kids, the rule it’s quite simple and understandable. The combination between the racing game and natural interface will lead to innovative application which is potential to be used as home therapy for post stroke patient. User will be required to move some parts of their body such as both hand, arm, head and even shoulder. The body tracking is provided by depth camera that can capture and interpret human body gesture recognition to be used as interaction command. The human skeleton will be displayed onto screen to control the car in the game and the result of racing will be synchronized with user excitement which is recorded through Kinect video.