Vocal Interface Research Articles

The present looks nothing like the Jetsons: Deceptive design in virtual assistants and the protection of the rights of users

The increasing popularity of virtual assistants (VAs) raises concerns about deceptive design (also referred to as dark patterns), that is, design tricks to influence users into buying or engaging more, hijacking their decision-making capability. The article argues that some recurring responses and prompts recited by VAs amount to deceptive design, matching some well-known dark patterns such as Price Comparison Prevention and Misdirection. It analyses the challenges of applying the EU consumer and personal data protection laws to deceptive design in VAs, exploring provisions on unfair practices and consumers’ rights (UCPD, CRD), fairness of personal data processing (GDPR), as well as the new rules on digital services (DSA) and the proposals for Data Act and AI Act. While the current legal framework offers a sufficient starting point to address the complexity and sophistication of deceptive design in virtual assistants, additional guidance on its application and interpretation is necessary to guarantee a high level of protection of the rights and interests of VA users. This article contributes to the ongoing debate on the regulation of deceptive design, and brings attention to the specific challenges deriving from virtual assistants due to the use of vocal interface, which can bring a paradigmatic shift from the legal perspective.

Efficient Self-Attention Model for Speech Recognition-Based Assistive Robots Control.

Assistive robots are tools that people living with upper body disabilities can leverage to autonomously perform Activities of Daily Living (ADL). Unfortunately, conventional control methods still rely on low-dimensional, easy-to-implement interfaces such as joysticks that tend to be unintuitive and cumbersome to use. In contrast, vocal commands may represent a viable and intuitive alternative. This work represents an important step toward providing a viable vocal interface for people living with upper limb disabilities by proposing a novel lightweight vocal command recognition system. The proposed model leverages the MobileNet2 architecture, augmenting it with a novel approach to the self-attention mechanism, achieving a new state-of-the-art performance for Keyword Spotting (KWS) on the Google Speech Commands Dataset (GSCD). Moreover, this work presents a new dataset, referred to as the French Speech Commands Dataset (FSCD), comprising 4963 vocal command utterances. Using the GSCD as the source, we used Transfer Learning (TL) to adapt the model to this cross-language task. TL has been shown to significantly improve the model performance on the FSCD. The viability of the proposed approach is further demonstrated through real-life control of a robotic arm by four healthy participants using both the proposed vocal interface and a joystick.

Design module for speech recognition graphical user interface browser to supports the web speech applications

The web speech API has made it possible to integrate audio data into web applications and make it a unique experience for all customers and users of modern applications. The website can only be accessed through devices equipped with a which stands for graphical user interface (GUI) and screen. For this to be done, there must be a physical attraction with such devices. This paper presents speech recognition using a web browser (SRWB) which permits browsing or surfing the internet with the use of a standard voice-only and vocal user interface (VUL) development. The SRWB system input from the users in form of vocal commands and covers these voice commands to HTTP requests. The SRWB system will send the voice commands to the web server for processing purposes and when the processing is done, the converted or translated HTTP response is outputted to the end-users in a voice format made audible with the attached loudspeakers. SAPI, developed by Microsoft, allows the use of SRWB in Windows applications. The algorithm is implemented by the system to achieve its goal for web content, classifying, analyzing, and sending important parts of web pages back to the end-user.

Proposed speech recognition interface framework using web browser

A website can only be accessed via devices that comes with both GUI which stands for Graphical User Interface and a screen. For this to be done, there must be physical attraction with such devices. This paper presents a speech recognition using web browser (SRWB) which permits browsing or surfing the Internet with the use of a standard voice only and VUL (Vocal User Interface development). The SRWB system input from the users in form of vocal commands and coverts these voice commands to HTTP requests. The SRWB system will sends the voice commands to the web server for processing purposes and when the processing is done, the converted or translated HTTP response is outputted to the end users in a voice format made audible with the attached loudspeakers. The system uses the Speech Application Programming Interface which is known in short as SAPI. SAPI which was developed by Microsoft allows SRWB to be used in the Windows applications. This algorithm is implemented by the system to achieve its target over content of the web to classify, analyse and send back important part of web pages to the end users. Speech recognition is used for controlling web browsers and applets, enhancing the internet, and filling out forms in numerous ways. Through speech synthesis, web pages are brought back to live, reduces audio that are sent over the internet. SRWB has many applications such as assisting the blind people to make use of the internet via speech and hearing. It will help both the young children and people with disability who can’t operate a keyboard to use the internet by just speaking. It grants enablement to people to interface the website contents through oral or audio commands instead of using a keyboard. For this system Microsoft access and SQL were used as databases while Visual Basic.NET was the programming environment used for developing and designing. SRWB is very easy to use, visually independent and can be used by anyone that is not computer literate.

Diabetic Foot Ulcer Treatment Device Using Peltier and Embedded Electronics

People’s degrading lifestyle reflects their health graphs and occurrence of different disease in their bodies. With different food habits and exercise patterns the endurance capacity of the body changes. Foot ulcer is a disease quite common today among Indians around. The foot ulcer creates a threat to the movement for human beings. It is a deep infected sore or damage in the foot sole commonly caused by nerve/skin damage. Several infections in the foot may also result in leg amputations. Foot ulcer is very common among diabetic patients. India is the country with the highest number of diabetics worldwide. Over 3 million people are infected with diabetes which is a big number to worry about. The CPR (Crude Prevalence Rate) for cities is around 9% of the population whereas in village areas the occurrence is around 3 percent [1]. Though the foot ulcer prevalence of diabetic patients in India is 3% which looks small but it is a big number. Our proposal is an embedded system innovation to develop a foot ulcer treatment system. It deals with the combined approach to embedded electronics and biomedical engineering to overcome the situation. We create a footrest equipped with Peltier, vibrator, and UV rays to overcome the issue. It is the first of its kind foot ulcer treatment equipment that can solve the issue. We also used the android technology to provide the vocal user interface to help the user. Clear instructions are displayed and spoken upon usage to the patients. The result obtained on the patients are highly acceptable and recommended for both urban and rural populations. The Atmega 328 P PU microcontroller is interfaced with the HC-05 Bluetooth and Vibrators through Relays to perform the task. The microcontroller acts as the overall think-tank to control and coordinate the overall system.

Sub-vocal speech pattern recognition of Hindi alphabet with surface electromyography signal

Summary Recently electromyography (EMG) based speech signals have been used as pattern recognition of phoneme, vocal frequency estimation, browser interface, and classification of speech related problem identification. Attempts have been made to use EMG signal for sub-vocal speech pattern recognition of Hindi phonemes and Hindi words. That provides the command sub-vocally to control the devices. Sub-vocal EMG data were collected from more than 10 healthy subjects aged between 25 and 30 years. EMG-based sub-vocal database are acquired from four channel BIOPAC MP-30 acquisition system. Four pairs of Ag-AgCl electrodes placed in the participant neck area of skin. AR coefficients and Cepstral coefficients were computed as features of EMG-based sub-vocal signal. Furthermore, these features are classified by HMM classifier. H2M MATLAB toolbox was used to develop HMM classifier for classification of phonemes. Results were averaged on 10 subjects. An average classification accuracy of Ka is found to be 85% whereas the classification accuracy of Kha and Gha is in between 88% and 90%. The classification accuracy rate of Ga was found to be 78% which was lesser as compared to Kha and Gha.

Text to Speech Conversion

The present paper has introduced an innovative, efficient and real-time cost beneficial technique that enables user to hear the contents of text images instead of reading through them. It combines the concept of Optical Character Recognition (OCR) and Text to Speech Synthesizer (TTS) in Raspberry pi. This kind of system helps visually impaired people to interact with computers effectively through vocal interface. Text Extraction from color images is a challenging task in computer vision. Text-to-Speech conversion is a method that scans and reads English alphabets and numbers that are in the image using OCR technique and changing it to voices. This paper describes the design, implementation and experimental results of the device. This device consists of two modules, image processing module and voice processing module. The device was developed based on Raspberry Pi v2 with 900 MHz processor speed.

Speaky for robots: the development of vocal interfaces for robotic applications

The currently available speech technologies on mobile devices achieve effective performance in terms of both reliability and the language they are able to capture. The availability of performant speech recognition engines may also support the deployment of vocal interfaces in consumer robots. However, the design and implementation of such interfaces still requires significant work. The language processing chain and the domain knowledge must be built for the specific features of the robotic platform, the deployment environment and the tasks to be performed. Hence, such interfaces are currently built in a completely ad hoc way. In this paper, we present a design methodology together with a support tool aiming to streamline and improve the implementation of dedicated vocal interfaces for robots. This work was developed within an experimental project called Speaky for Robots. We extend the existing vocal interface development framework to target robotic applications. The proposed solution is built using a bottom-up approach by refining the language processing chain through the development of vocal interfaces for different robotic platforms and domains. The proposed approach is validated both in experiments involving several research prototypes and in tests involving end-users.

An Empirical Evaluation of a Vocal User Interface for Programming by Voice

Although Graphical User Interfaces (GUIs) often improve usability, individuals with physical disabilities may be unable to use a mouse and keyboard to navigate through a GUI-based application. In such situations, a Vocal User Interface (VUI) may be a viable alternative. Existing vocal tools (e.g., Vocal Joystick) can be integrated into software applications; however, integrating an assistive technology into a legacy application may require tedious and manual adaptation. Furthermore, the challenges are deeper for an application whose GUI changes dynamically (e.g., based on the context of the program) and evolves with each new application release. This paper provides a discussion of challenges observed while mapping a GUI to a VUI. The context of the authors' examples and evaluation are taken from Myna, which is the VUI that is mapped to the Scratch programming environment. Initial user studies on the effectiveness of Myna are also presented in the paper.

An integrated system for voice command recognition and emergency detection based on audio signals

The recent reports on population ageing in the most advanced countries are driving governments and the scientific community to focus on technologies for providing assistance to people in their own homes. Particular attention has been devoted to solutions based on acoustic signals since they provide a convenient way to monitor people activities and they enable hands-free human–machine interfaces. In this context, this paper presents a complete solution for voice command recognition and emergency detection based on audio signals entirely integrated in a low-consuming embedded platform. The system combines an active operation mode were distress calls are captured and a vocal interface is enabled for controlling the home automation subsystem, and a pro-active mode, were a novelty detection algorithm detects abnormal acoustic events to alert the user of a possible emergency. In the first operation mode, a Voice Activity Detector captures voice segments of the audio signal, and a speech recogniser detects commands and distress calls. In the pro-active mode, an acoustic novelty detector is employed in order to be able to deal with unknown sounds, thus not requiring an explicit modelling of emergency sounds. In addition, the system integrates a VoIP infrastructure so that emergencies can be communicated to relatives or care centres. The monitoring unit is equipped with multiple microphones and it is connected to the home local area network to communicate with the home automation subsystem. The algorithms have been implemented in a low-consuming embedded platform based on a ARM Cortex-A8 CPU. The effectiveness of the adopted algorithms has been tested on two different databases: ITAAL and A3Novelty. The obtained results show that the adopted solutions are suitable for speech and audio event monitoring in a realistic scenario.

The self-taught vocal interface

Speech technology is firmly rooted in daily life, most notably in command-and-control (C&C) applications. C&C usability downgrades quickly, however, when used by people with non-standard speech. We pursue a fully adaptive vocal user interface (VUI) which can learn both vocabulary and grammar directly from interaction examples, achieving robustness against non-standard speech by building up models from scratch. This approach raises feasibility concerns on the amount of training material required to yield an acceptable recognition accuracy. In a previous work, we proposed a VUI based on non-negative matrix factorisation (NMF) to find recurrent acoustic and semantic patterns comprising spoken commands and device-specific actions, and showed its effectiveness on unimpaired speech. In this work, we evaluate the feasibility of a self-taught VUI on a new database called domotica-3, which contains dysarthric speech with typical commands in a home automation setting. Additionally, we compare our NMF-based system with a system based on Gaussian mixtures. The evaluation favours our NMF-based approach, yielding feasible recognition accuracies for people with dysarthric speech after a few learning examples. Finally, we propose the use of a multi-layered semantic frame structure and demonstrate its effectiveness in boosting overall performance.

Fast vocabulary acquisition in an NMF-based self-learning vocal user interface

In command-and-control applications, a vocal user interface (VUI) is useful for handsfree control of various devices, especially for people with a physical disability. The spoken utterances are usually restricted to a predefined list of phrases or to a restricted grammar, and the acoustic models work well for normal speech. While some state-of-the-art methods allow for user adaptation of the predefined acoustic models and lexicons, we pursue a fully adaptive VUI by learning both vocabulary and acoustics directly from interaction examples. A learning curve usually has a steep rise in the beginning and an asymptotic ceiling at the end. To limit tutoring time and to guarantee good performance in the long run, the word learning rate of the VUI should be fast and the learning curve should level off at a high accuracy. In order to deal with these performance indicators, we propose a multi-level VUI architecture and we investigate the effectiveness of alternative processing schemes. In the low-level layer, we explore the use of MIDA features (Mutual Information Discrimination Analysis) against conventional MFCC features. In the mid-level layer, we enhance the acoustic representation by means of phone posteriorgrams and clustering procedures. In the high-level layer, we use the NMF (Non-negative Matrix Factorization) procedure which has been demonstrated to be an effective approach for word learning. We evaluate and discuss the performance and the feasibility of our approach in a realistic experimental setting of the VUI-user learning context.

OMAS—a flexible multi-agent environment for CSCWD

CSCW requires efficient support to distributed teams. Several technologies seem to be good candidates for supporting CSCW among which groupware or agents. Groupware is efficient but rather inflexible, multi-agent systems are more flexible but difficult to implement. OMAS (Open Multi-Agent System) is a multi-agent platform derived from a number of previous research projects and teaching courses. With OMAS we tried to answer the problems of architecture, skill definition, protocol implementation, ontologies, dialog using natural language, and debugging, so as to simplify the work of an application designer. OMAS uses a “plug in” type approach offering a model of generic cognitive agent. Agents are of three types: service, personal assistants and transfer. They are organized as coteries, meaning that each agent sees all messages. Agents have separate ontologies and tasks models. Agents are multi-threaded and can execute several tasks concurrently. Personal assistants have prototype dialogs and vocal interface. OMAS has been designed for supporting CSCWD projects or applications. It can be downloaded from the Internet.

CityMobil

There are several human factor concerns with highly autonomous or semiautonomous driving, such as transition of control, loss of skill, and dealing with automated system errors. Four CityMobil experiments studied the eLane concept for dual-mode cars, and the results of one are described. The open eLane concept brings together road infrastructure and technical developments in vehicle automation to allow automated driving. The goal for the driving simulator experiment was to design and test the difference between a vocal and an acoustic user interface for a dual-mode vehicle driven both manually and automatically. In the experiment ( n = 24), driver behavior was observed with a focus on the transition of control and the occurrence of system errors. Performance of transition of control was adequate for both interfaces at the beginning and end of an eLane. In the case of system failure, 15% of drivers failed to take control of the car in time for both interfaces. However, of those who did regain control, drivers with the vocal interface were faster. Moreover, a subjective questionnaire showed that the vocal interface was perceived as more positive than the acoustic interface. The study suggests that the vocal interface was preferred by participants and can be recommended for the human–machine interface of dual-mode vehicles, especially for providing warnings about system malfunctioning.

Speech recognition for the anaesthesia record during crisis scenarios

Introduction This article describes the evaluation of a prototype speech-input interface to an anaesthesia patient record, conducted in a full-scale anaesthesia simulator involving six doctor–nurse anaesthetist teams. Objective The aims of the experiment were, first, to assess the potential advantages and disadvantages of a vocal interface compared to the traditional touch-screen and keyboard interface to an electronic anaesthesia record during crisis situations; second, to assess the usability in a realistic work environment of some speech input strategies (hands-free vocal interface activated by a keyword; combination of command and free text modes); finally, to quantify some of the gains that could be provided by the speech input modality. Methods Six anaesthesia teams composed of one doctor and one nurse were each confronted with two crisis scenarios in a full-scale anaesthesia simulator. Each team would fill in the anaesthesia record, in one session using only the traditional touch-screen and keyboard interface while in the other session they could also use the speech input interface. Audio-video recordings of the sessions were subsequently analysed and additional subjective data were gathered from a questionnaire. Analysis of data was made by a method inspired by queuing theory in order to compare the delays associated to the two interfaces and to quantify the workload inherent to the memorisation of items to be entered into the anaesthesia record. Results The experiment showed on the one hand that the traditional touch-screen and keyboard interface imposes a steadily increasing mental workload in terms of items to keep in memory until there is time to update the anaesthesia record, and on the other hand that the speech input interface will allow anaesthetists to enter medications and observations almost simultaneously when they are given or made. The tested speech input strategies were successful, even with the ambient noise. Speaking to the system while working appeared feasible, although improvements in speech recognition rates are needed. Conclusion A vocal interface leads to shorter time between the events to be registered and the actual registration in the electronic anaesthesia record; therefore, this type of interface would likely lead to greater accuracy of items recorded and a reduction of mental workload associated with memorisation of events to be registered, especially during time constrained situations. At the same time, current speech recognition technology and speech interfaces require user training and user dedication if a speech interface is to be used successfully.

Making non-fluent aphasics speak: sing along!

A classic observation in neurology is that aphasics can sing words they cannot pronounce otherwise. To further assess this claim, we investigated the production of sung and spoken utterances in eight brain-damaged patients suffering from a variety of speech disorders as a consequence of a left-hemisphere lesion. In Experiment 1, the patients were tested in the repetition and recall of words and notes of familiar material. Lyrics of familiar songs, as well as words of proverbs and prayers, were not better pronounced in singing than in speaking. Notes were better produced than words. In Experiment 2, the aphasic patients repeated and recalled lyrics from novel songs. Again, they did not produce more words in singing than in speaking. In Experiment 3, when allowed to sing or speak along with an auditory model while learning novel songs, aphasics repeated and recalled more words when singing than when speaking. Reduced speed or shadowing cannot account for this advantage of singing along over speaking in unison. The results suggest that singing in synchrony with an auditory model--choral singing--is more effective than choral speech, at least in French, in improving word intelligibility because choral singing may entrain more than one auditory-vocal interface. Thus, choral singing appears to be an effective means of speech therapy.

Evaluation of a Voice Interface Management System

The final prototype of a centralized voice information system, developed under the framework of the European CeMVocAS (Centralised Management of Vocal Interfaces Aiming at a Better Automotive Safety) project, was evaluated. This system manages all types of voice messages to the driver (i.e., input and output voice flows to and from car telephone, radio, and driver assistance systems) to minimize their disruptive impact on the basic task of driving. Analysis of the results included evaluation of ( a) the part of the system responsible for determining the driver’s potential availability to receive voice messages [driving situation recognition unit (DSRU)] and ( b) the complete system (DSRU plus centralized voice interface management system). Results indicate that the system is quite robust in terms of vehicle, driver, and route variability; in addition, levels of system effectiveness, system efficiency, and user satisfaction were high.

A59 Development of a software using a touch tone phone system and a vocal interface for automatic registration and randomization in multicenter clinical trials: Further experience of the European lung cancer working party

When foraging, male big brown bats produce ultrasonic social calls. The calls repel rival bats from the caller and its prey, and increase the caller's foraging success during their high-speed aerial excursions.

An application of speech recognition and synthesis

This investigation has produced a bidirectional vocal interface between man and machine to control an electropneumatic robotic device. This was achieved using low-cost speech input/output boards commercially available for the Apple II microcomputer, and interfacing these, through the Apple and a Motorola M6800 D2 Kit, to the electropneumatic device. The full capabilities of the Heuristics Speechlink recognition unit and the Mountain Hardware Supertalker were used, to provide a very user friendly system which automatically compensated for vocal drift due to stress or illness.