Transforming graphical interfaces into auditory interfaces

Abstract

This work explores transforming graphical interfaces into auditory interfaces. By transparently transforming graphical application written with a commercial toolkit, I have been able to investigate strategies for generically modeling graphical interfaces. I use hierarchical structures to model graphical interfaces since these models best captured abstract representations of the graphical interfaces. These logical representations match how blind users conceptualize graphical interfaces, as well as correspond to the components that sighted users perceive. Additionally, creating usable auditory interfaces based on these models is feasible. I have helped build the Mercator system that transparently creates hierarchical models of graphical interfaces as the applications are executing. With generic models of graphical interfaces, the next task has been designing an interface that utilized only auditory output. Coupled with keyboard input, the Mercator interface allows users to explore and manipulate auditory representations of graphical interfaces. The contents of the interface is conveyed using auditory icons. These nonspeech sounds represent the identity and attributes of interface objects by mimicking real world sounds of the interface structure. This navigation strategy reinforces a blind user's mental model of the interface while providing a consistent set of controls for accessing all objects in the interface. The decision to use auditory icons in the interface spurred questions in designing these cues. By investigating how people described everyday sounds, and how they expected everyday sounds to be used in graphical interfaces, I highlighted several guidelines for designing auditory icons. These guidelines informed the selection of cues to be used in Mercator. My focus on blind users positively influenced my research in two ways. First, their requirements for access to graphical interfaces forced me to investigate generic transformations where the graphical and auditory interfaces shared an underlying representation. The most challenging aspect of the design was facilitating collaboration between sighted and blind users. Second, their real need for access technology has led me to help build a robust system that supports access to numerous applications and levels of user expertise.