A Physical User Interface Description Language for Interface Digital Twins

Abstract

Although the American Disability Act defines guidelines and regulations in terms of equitable access for digital and physical products, the accessibility divide between hardware and software technology is increasing. In the last decade, accessibility has received increasing attention in software development and, particularly, smartphone applications and web design. On the contrary, the design of physical products such as home appliances, electronic devices, and vending machines, lacks compliance with most basic accessibility rules, which impacts the quality of life of people with disabilities and, specifically, individuals who are blind or have cognitive conditions. As a result, people with disabilities are unable to independently learn and use several physical products and devices, due to the lack of accessibility of their User Interface (UI).Our previous work introduced the concept of Interface Digital Twin (IDT), that is, the digital replica of a physical UI rendered in the form of an interactive model: IDTs leverage the features of the smartphone (e.g., text-to-speech, vibration, and the possibility of adding more information) to increase the accessibility of the interface of a physical device (e.g., translate visual labels into speech), help individuals understand its components and learn how to operate it (e.g., interactive user manual), and support users in accomplishing their tasks on the actual device (e.g., provide step-by-step guidance). Our research demonstrated that IDTs can be introduced without requiring any modifications to currently available technology, which makes them suitable for enhancing the accessibility of existing devices, especially in the case of individuals with sensory or cognitive disabilities. Furthermore, IDT can be utilized to and digitize the functionality of physical devices. For instance, in the case of a ticket vending machine, the entire process can be realized digitally, using a simplified interface for buying the ticket on the smartphone. By doing this, we aim at rendering physical devices a “physical tag” for digital processes, which would help transition to a more paperless and digital society.In this paper, we detail a Physical User Interface Description Language (PUIDL) that has the purpose of representing the main characteristics of the UI of a physical device into a digital format so that it can be utilized to create IDTs on smartphones and other personal devices. We introduce the language and we highlight the differences between the proposed PUIDL and existing User Interface Description Languages, which are designed to represent software interfaces, only. We detail the main characteristics of the language, we discuss its syntax and notation, and we demonstrate how the our system can be utilized to implement the IDT of an actual physical device. Furthermore, we analyze the effectiveness, modularity, interoperability, and performance of the proposed language in describing physical user interfaces. Finally, we describe how PUIDLs can support the use IDTs for controlling physical devices in the context of the Internet of Things (IoT).