Auditory Interface Research Articles

Ensemble Convoluted Feature Extraction for Affective Auditory P300 Brain-Computer Interfaces.

Since most of P300 brain-computer interface (BCI) methods have assumed visual events, they are not always suitable for the BCI with auditory events, and feature extraction methods appropriate for auditory P300 BCI are required. This study proposed ensemble convoluted feature extraction for affective auditory P300 BCI, which took advantage of auditory responses elicited by different affective sounds. The proposed method was compared to feature extraction method that uses the canonical correlation analysis in addition to the traditional method. Those methods were evaluated on the dataset recorded from the improved affective auditory P300 BCI system. The mean online classification accuracy was 84.1% when using the traditional feature. The offline analysis showed that the proposed ensemble convolution feature extraction achieved significantly higher accuracy (86.75%) than the traditional method. The propose feature extraction may effective for the multi-channel time-series BCI that is featured by different stimuli.

Teleoperated path following and trajectory tracking of unmanned vehicles using spatial auditory guidance system

The majority of Human–Machine Interfaces (HMIs) designed for the teleoperation of unmanned vehicles only present information visually. Frequent overloading of the operator’s visual channel may result in accidents. To reduce the occurrence of accidents and improve overall operating performance, this paper proposes the extension of the HMI to the hearing modality. The proposed HMI takes the form of a spatial auditory display for complex navigation tasks such as path following (route guidance) or trajectory tracking. The interface incorporates two novel components. The first is the application of a modified “lookahead distance” guidance strategy, as a guidance law for generating a reference. The reference is presented to the operator as a spatial auditory image of the virtual target to be followed. Before being presented to the operator, the reference information is preconditioned through “supernormal” transformations to increase azimuth and distance resolution in regions of interest and thus to improve efficiency and overcome the comparative disadvantages of the hearing channel. A series of experiments show that the proposed guidance strategy provides a comprehensible and effective auditory reference. The enhanced supernormal auditory interface improves performance by reducing the tracking error of both path following and trajectory tracking compared to a non-enhanced interface without the supernormal transformation. Experiment participants report that the novel auditory interface is intuitive and easy to use, providing very good situational awareness. While the auditory interface investigated in this paper is shown to be effective for the teleoperation of unmanned vehicles, it may have other applications such as navigation for the visually impaired.

A Vision-Based Wayfinding System for Visually Impaired People Using Situation Awareness and Activity-Based Instructions.

A significant challenge faced by visually impaired people is ‘wayfinding’, which is the ability to find one’s way to a destination in an unfamiliar environment. This study develops a novel wayfinding system for smartphones that can automatically recognize the situation and scene objects in real time. Through analyzing streaming images, the proposed system first classifies the current situation of a user in terms of their location. Next, based on the current situation, only the necessary context objects are found and interpreted using computer vision techniques. It estimates the motions of the user with two inertial sensors and records the trajectories of the user toward the destination, which are also used as a guide for the return route after reaching the destination. To efficiently convey the recognized results using an auditory interface, activity-based instructions are generated that guide the user in a series of movements along a route. To assess the effectiveness of the proposed system, experiments were conducted in several indoor environments: the sit in which the situation awareness accuracy was 90% and the object detection false alarm rate was 0.016. In addition, our field test results demonstrate that users can locate their paths with an accuracy of 97%.

Brain-Computer Interface application: auditory serial interface to control a two-class motor-imagery-based wheelchair

BackgroundCertain diseases affect brain areas that control the movements of the patients’ body, thereby limiting their autonomy and communication capacity. Research in the field of Brain-Computer Interfaces aims to provide patients with an alternative communication channel not based on muscular activity, but on the processing of brain signals. Through these systems, subjects can control external devices such as spellers to communicate, robotic prostheses to restore limb movements, or domotic systems. The present work focus on the non-muscular control of a robotic wheelchair.MethodA proposal to control a wheelchair through a Brain–Computer Interface based on the discrimination of only two mental tasks is presented in this study. The wheelchair displacement is performed with discrete movements. The control signals used are sensorimotor rhythms modulated through a right-hand motor imagery task or mental idle state. The peculiarity of the control system is that it is based on a serial auditory interface that provides the user with four navigation commands. The use of two mental tasks to select commands may facilitate control and reduce error rates compared to other endogenous control systems for wheelchairs.ResultsSeventeen subjects initially participated in the study; nine of them completed the three sessions of the proposed protocol. After the first calibration session, seven subjects were discarded due to a low control of their electroencephalographic signals; nine out of ten subjects controlled a virtual wheelchair during the second session; these same nine subjects achieved a medium accuracy level above 0.83 on the real wheelchair control session.ConclusionThe results suggest that more extensive training with the proposed control system can be an effective and safe option that will allow the displacement of a wheelchair in a controlled environment for potential users suffering from some types of motor neuron diseases.

Auditory User Interface Guideline Development for Industrial Sound Design: focused on Function Preference and Sexual Difference

Auditory User Interface Guideline Development for Industrial Sound Design: focused on Function Preference and Sexual Difference

Biomusic: An Auditory Interface for Detecting Physiological Indicators of Anxiety in Children.

For children with profound disabilities affecting communication, it can be extremely challenging to identify salient emotions such as anxiety. If left unmanaged, anxiety can lead to hypertension, cardiovascular disease, and other psychological diagnoses. Physiological signals of the autonomic nervous system are indicative of anxiety, but can be difficult to interpret for non-specialist caregivers. This paper evaluates an auditory interface for intuitive detection of anxiety from physiological signals. The interface, called “Biomusic,” maps physiological signals to music (i.e., electrodermal activity to melody; skin temperature to musical key; heart rate to drum beat; respiration to a “whooshing” embellishment resembling the sound of an exhalation). The Biomusic interface was tested in two experiments. Biomusic samples were generated from physiological recordings of typically developing children (n = 10) and children with autism spectrum disorders (n = 5) during relaxing and anxiety-provoking conditions. Adult participants (n = 16) were then asked to identify “anxious” or “relaxed” states by listening to the samples. In a classification task with 30 Biomusic samples (1 relaxed state, 1 anxious state per child), classification accuracy, sensitivity, and specificity were 80.8% [standard error (SE) = 2.3], 84.9% (SE = 3.0), and 76.8% (SE = 3.9), respectively. Participants were able to form an early and accurate impression of the anxiety state within 12.1 (SE = 0.7) seconds of hearing the Biomusic with very little training (i.e., < 10 min) and no contextual information. Biomusic holds promise for monitoring, communication, and biofeedback systems for anxiety management.

MobiLAudio – a multimodal content delivery platform for geo-services

Delivering high-quality context-relevant information in a timely manner is a priority for location-based services (LBS) where applications require an immediate response based on spatial interaction. Previous work in this area typically focused on ever more accurately determining this interaction and informing the user in the customary graphical way using the visual modality. This paper describes the research area of multimodal LBS and focuses on audio as the key delivery mechanism. This new research extends familiar graphical information delivery by introducing a geoservices platform for delivering multimodal content and navigation services. It incorporates a novel auditory user interface (AUI) that enables delivery of natural language directions and rich media content using audio. This unifying concept provides a hands-free modality for navigating a mapped space while simultaneously enjoying rich media content that is dynamically constructed using such mechanisms as algorithmic music and phrase synthesis to generate task-relevant content based on the path taken. This paper outlines the innovative ideas employed in the design of the AUI and details the geoservices platform developed for facilitating the authoring and delivery of multimodal LBS applications. The paper concludes with a discussion on the results of a live user trial. The results are analysed and presented to validate the original hypothesis for this approach, address the research questions outlined and to inform further research directions. The results show that the proposed solution significantly progresses the state-of-the-art in terms of mobile tour production. The results also show that an AUI is an effective modality for the delivery of audio content and natural directions when used in combination with a graphical user interface, producing significantly reduced overheads in terms of content size and network usage. The results also indicate that the AUI provides a good overall user experience, performing well in the user trial.

Teleoperated trajectory tracking of remotely operated vehicles using spatial auditory interface

Abstract: The majority of Human-Machine-Interfaces (HMIs) designed for teleoperation of Unmanned Vehicles (UVs) present information only visually. Frequent overloading of operator’s visual channel may cause unwanted mishaps. In order to reduce such mishaps and improve overall operating performance, this paper proposes the extension of the HMI by hearing modality in the form of spatial auditory display for trajectory tracking, the most complex guidance task. The main novelty of the interface is introduction of guidance laws to generate the reference presented to the operator as a spatial auditory image of the virtual target to be followed. Guidance laws for teleoperated trajectory tracking are based on modified lookahead distance strategy, known from path following applications. The paper also analyzes the stability of the kinematic controller based on this new guidance control law. The experiments show that the novel guidance strategy provides comprehensible and effective reference providing excellent trajectory tracking performance.

Pursuit : A Foraging Simulation Tool for Research and Teaching.

This article describes a software tool called "Pursuit" that is intended to be used for both research and teaching on the topic of optimal foraging theory. The tool provides a dynamic graphical and auditory interface in which users encounter different prey animals and then must decide whether to pursue or ignore the encountered prey. Based on the characteristics of the prey in the foraging environment and the decisions of the player, each user harvests a set of prey per round and achieves a corresponding foraging return rate. Administrators of Pursuit specify the environmental parameters that determine what prey users will encounter. All environmental parameters and user decisions are tracked and logged for analysis. We created this tool for laboratory experiments, but we believe Pursuit could also be an engaging and effective teaching tool, whereby students adopt the role of forager, and through such play, experience a simulated foraging context and learn about foraging theory. Pursuit is freely available and can run on any platform that supports Java, including Mac OS, Windows, and Linux.

Development of auditory design guidelines for improving learning on mobile phones

Although auditory information in mobile learning (m-learning) can be an important resource for delivering knowledge and information, the importance of guidelines for designing auditory instructions has largely been overlooked. Therefore, this study considered the characteristics of m-learning, mobility and personalization, with the aim of developing and validating auditory design guidelines to improve learning on mobile phones. The present study covers m-learning with cellphones only. Auditory design guidelines were developed and revised using an iterative educational design research process. The auditory design guidelines were developed by analyzing previous studies related to m-learning and auditory interface design, and revised using three validation methods. The expert reviews and usability evaluations were conducted as an internal validation method, and a field evaluation was used as an external validation method to confirm the feasibility and educational effects. The results showed that the auditory design guidelines could be applied effectively to the design of audio instructions delivered on smart phones. Finally, a total of ten guidelines were developed, three for auditory-only design, four for attention design, and three for personalization design. This paper discusses the theoretical, empirical and practical aspects of the design guidelines presented.

Sonification of guidance data during road crossing for people with visual impairments or blindness

In the last years several solutions were proposed to support people with visual impairments or blindness during road crossing. These solutions focus on computer vision techniques for recognizing pedestrian crosswalks and computing their relative position from the user. Instead, this contribution addresses a different problem; the design of an auditory interface that can effectively guide the user during road crossing. Two original auditory guiding modes based on data sonification are presented and compared with a guiding mode based on speech messages.Experimental evaluation shows that there is no guiding mode that is best suited for all test subjects. The average time to align and cross is not significantly different among the three guiding modes, and test subjects distribute their preferences for the best guiding mode almost uniformly among the three solutions. From the experiments it also emerges that higher effort is necessary for decoding the sonified instructions if compared to the speech instructions, and that test subjects require frequent 'hints' (in the form of speech messages). Despite this, more than 2/3 of test subjects prefer one of the two guiding modes based on sonification. There are two main reasons for this: firstly, with speech messages it is harder to hear the sound of the environment, and secondly sonified messages convey information about the quantity of the expected movement. HighlightsWe study audio interfaces to guide people with visual impairment in road crossing.We design two sonifications and implement them in a mobile prototype.We empirically compare the sonifications with a speech-based interface.The two sonifications are less intuitive but effective even after little training.Most subjects (75%) declare to prefer one of the two sonifications over speech.

The anterior contralateral response improves performance in a single trial auditory oddball BMI

Abstract Auditory Brain Machine Interfaces were designed for patients with severe neurofunctional disabilities, such as those suffering with late-stage amyotrophic lateral sclerosis (ALS), who have impaired eye movements or are unable to maintain gaze preventing them from using visual strategies for communication. This study explores the three-stimulus auditory oddball paradigm with binary choices (yes/no) associated with Empirical Mode Decomposition to extract features used to train and test a Support Vector Machine classifier. Data from standard EEG channels and from the N200-anterior-contralateral (N2ac) response signal were tested. Ten healthy male subjects, age 20 to 27 years, participated in the experiment. The best performance (average classification accuracy of 87.41% and information transfer ratio of 6.48 bit/min) was achieved when features extracted from the N2ac response were added to features extracted from the EEG channels. Also, the results showed that by using target stimuli with larger frequency separation helps the subjects focus better on the desired answer.

위치 인식 기반 시스템을 활용한 체험형 공간의 청각적 인터페이스 연구 - 소릿길 프로젝트 GPS Sound Mapping System 기반으로

위치 인식 기반 시스템을 활용한 체험형 공간의 청각적 인터페이스 연구 - 소릿길 프로젝트 GPS Sound Mapping System 기반으로

The design of an in-vehicle assistance system to support eco-driving

This driving simulator study was the second of two studies investigating the most effective and acceptable in-vehicle system for the provision of guidance on fuel efficient accelerator usage. Three eco-driving interfaces were selected for test (a second-order display visual display with auditory alerts and two haptic accelerator pedal systems) following a pilot study of 12 different interfaces. These systems were tested in a range of eco-driving scenarios involving acceleration, deceleration and speed maintenance, and assessed through their effects on fuel economy, vehicle control, distraction, and driver subjective feedback. The results suggest that a haptic accelerator pedal system is most effective for preventing over-acceleration, whilst minimal differences were observed between systems in terms of the effect of the assistance provided to prevent under-acceleration. The visual–auditory interface lowered the time spent looking towards the road, indicating a potential negative impact on driver safety from using this modality to provide continuous green driving support. Subjective results were consistent with the objective findings, with haptic pedal systems creating lower perceived workload than a visual–auditory interface. Driver acceptability ratings suggested a slight favouring of a haptic-force system for its usefulness, whereas the more subtle haptic-stiffness system was judged more acceptable to use. These findings offer suggestions for the design of a user-friendly, eco-driving device that can help drivers improve their fuel economy, specifically through the provision of real-time guidance on the manipulation of the accelerator pedal position.

Music Training Interface for Visually Impaired through a Novel Approach to Optical Music Recognition

AbstractSome inherited barriers which limits the human abilities can be surprisingly win through technology. This research focuses on defining a more reliable and a controllable interface for visually impaired people to read and study eastern music notations which are widely available in printed format. One of another concept behind was that differently-abled people should be assisted in a way which they can proceed interested tasks in an independent way. The research provide means to continue on researching the validity of using a controllable auditory interface instead using Braille music scripts converted with the help of 3rd parties. The research further summarizes the requirements aroused by the relevant users, design considerations, evaluation results on user feedbacks of proposed interface.

Auditory interface for teleoperation - Path following experimental results

For teloperated unmanned vehicles, mishaps tend to occur during the periods of high workload, in situations where the operator must perform complex and stressful tasks. However, with many tasks performed simultaneously with flying, the relevant information is typically dispersed on a number of screens overloading the operator's visual channel. In order to address these unique human-factors problems associated with unmanned vehicles we suggest the use of an auditory display as a mean to reduce visual workload, to enhance situation awareness, and mitigate the visual and cognitive demands of contemporary marine teleoperations. Experiments were performed on the remotely operated surface marine platform (PlaDyPos) developed at the Laboratory for Underwater Systems and Technologies, Faculty of Electrical Engineering and Computing, University of Zagreb. The results show that the concept, guidance-by-sound is feasible in the real environment. The results are in line with previously obtained results from the real-time simulator showing that tracking quality can be further improved introducing supernormal auditory cues in order to provide better-than-normal operator's auditory resolution in the frontal region. We conclude that the use of hearing in the form of the auditory display emerges as an important advantage. Since practice has a major effect on performance, there is definitely more room for improvement in using interfaces we are not trained for.

Overview of auditory representations in human-machine interfaces

In recent years, a large number of research projects have focused on the use of auditory representations in a broadened scope of application scenarios. Results in such projects have shown that auditory elements can effectively complement other modalities not only in the traditional desktop computer environment but also in virtual and augmented reality, mobile platforms, and other kinds of novel computing environments. The successful use of auditory representations in this growing number of application scenarios has in turn prompted researchers to rediscover the more basic auditory representations and extend them in various directions. The goal of this article is to survey both classical auditory representations (e.g., auditory icons and earcons) and those auditory representations that have been created as extensions to earlier approaches, including speech-based sounds (e.g., spearcons and spindex representations), emotionally grounded sounds (e.g., auditory emoticons and spemoticons), and various other sound types used to provide sonifications in practical scenarios. The article concludes by outlining the latest trends in auditory interface design and providing examples of these trends.

An Empirical Analysis of Auditory Interfaces in Human-computer Interaction

This study attempted to compare usability of auditory interfaces, which is a comprehensive concept that includes safety, utility, effectiveness, and efficiency, in personal computing environments: verbal messages (speech sounds), earcons (musical sounds), and auditory icons (natural sounds). This study hypothesized that verbal messages would offer higher usability than earcons and auditory icons, since the verbal messages are easy to interpret and understand based on semiotic process. In this study, usability was measured by a set of seven items: ability to inform what the program is doing, relevance to visual interfaces, degree of stimulation, degree of understandability, perceived time pressure, clearness of sound outputs, and degrees of satisfaction. Through the experimental research, the results showed that verbal messages provided the highest level of usability. On the contrary, auditory icons showed the lowest level of usability, as they require users to establish new coding schemes, and thus demand more mental effort from users.

A Sample Work Developed Using Speech Recognition Technology about basic Language Education: Speak and Learn – Learn with Speech Recognition

Click In contrast to common language teaching software, a software has been developed, using speech recognition technology with a rich visual and auditory interface allowing user to pronounce and learn pronunciation activities with their pictures, in this work. Purpose is to provide material for children in early age and special education centers to contribute language education. In this way, the usage of learning techniques in language teaching in a computer aided environment to learn pronunciations and some other concept with users’ speech will be possible.

Eyes-free interaction with free-hand gestures and auditory menus

Auditory interfaces can overcome visual interfaces when a primary task, such as driving, competes for the attention of a user controlling a device, such as radio. In emerging interfaces enabled by camera tracking, auditory displays may also provide viable alternatives to visual displays. This paper presents a user study of interoperable auditory and visual menus, in which control gestures remain the same in the visual and the auditory domain. Tested control methods included a novel free-hand gesture interaction with camera-based tracking, and touch screen interaction with a tablet. The task of the participants was to select numbers from a visual or an auditory menu including a circular layout and a numeric keypad layout. Results show, that even with participant's full attention to the task, the performance and accuracy of the auditory interface are the same or even slightly better than the visual when controlled with free-hand gestures. The auditory menu was measured to be slower in touch screen interaction, but questionnaire revealed that over half of the participants felt that the circular auditory menu was faster than the visual menu. Furthermore, visual and auditory feedback in touch screen interaction with numeric layout was measured fastest, touch screen with circular menu second fastest, and the free-hand gesture interface was slowest. The results suggest that auditory menus can potentially provide a fast and desirable interface to control devices with free-hand gestures.