Graphical User Interface Objects Research Articles

Deep learning‐based prototyping of android GUI from hand‐drawn mockups

Recently, transforming graphical user interface (GUI) mockups into code becomes a common challenging practice for current software developers. However, this transformation usually takes time especially when GUI changes keep pace with evolutionary features. There are many studies admitted this challenge and presented solutions in terms of computer-based GUI mockups. However, there is a research gap in this kind of research as very few of them adopted hand-drawn mockups as an input. In this study, the authors employed YOLOv5 is a fast and accurate deep learning framework to automate the process of converting hand-drawn GUI mockups into Android-based GUI prototype. The process starts with detecting all GUI mockups in an input image and determining their bounding boxes, classifying these mockups into their corresponding GUI objects, then finally aligning these objects together to form the output prototype based on the layout presented in the input image. Experimental results show the effectiveness of the proposed approach in generating a visually appealing Android GUI from hand-drawn mockups with a recognition accuracy of 98.54% when tested on various hand-drawn GUI structures designed by five developers.

Adaptive Color Selection to Limit Power Consumption for Multi-Object GUI Applications in OLED-Based Mobile Devices

Recently, as OLED displays are widely used in smartphones, their power management has become an important issue. Since OLED display devices show a large difference in power consumption depending on the colors used, proper color selection is important to reduce power consumption, but this may affect quality-of-experience (QoE) perceived by users. To address this tradeoff, this paper proposes a new power management scheme for graphical-user-interface (GUI) applications in OLED-based mobile devices. First, we examine the relationship between power consumption and color usage based on actual measurements and derive a model to express color preferences. Next, we present a dynamic programming algorithm that adaptively changes the color of the GUI objects in the applications to limit the total power consumption while effectively reflecting the color preferences of the user. The proposed scheme was implemented on a commercial smartphone, and real power measurements were conducted to examine whether it uses the given energy budget accurately under various color preference configurations. Experimental results show that color preferences significantly affect both power consumption and QoE, thereby demonstrating that personalized color selection plays an important role in reducing total power consumption of an OLED smartphone.

Lessons learned from automating tests for an operations support system

AbstractWe present experience gained in automating tests for an operations support system. A major portion of the effort was devoted to extending a commercial test tool so that testers could easily manipulate graphical user interface (GUI) objects on two implementations of the application. For this purpose, we developed a test automation library as support infrastructure for writing tests. The challenges and tradeoffs are discussed such as simplicity/complexity for a tester versus a library developer, hiding/exposing window hierarchy to the tester, providing common methods for different types of GUI objects, transparently manipulating custom GUI widgets, and coping with data‐dependent test cases. We discuss the requirements of test code reusability, maintainability, and portability, and describe the solutions we found. In addition, we offer observations about benefits and pitfalls of test automation, our recommendations for maximizing return on investment, and results from automating a variety of tests. Copyright © 2002 John Wiley & Sons, Ltd.

GUI object search method using a tactile display

As a means for the visually impaired to operate graphical user interface (GUI) applications, screen-reader software that combines keypad operation and a speech synthesizer is emerging. When keypad operation is used, however, the search for GUI objects requires more time compared to mouse operation. In other words, the difference between the sighted and the visually impaired in information access has gradually been enhanced with the advent of GUI, and it is desired to develop a system by which the visually impaired can operate GUIs comfortably. From such a viewpoint, the authors propose a nonvisual representation of the GUI object, by representing the hierarchy and the functions of GUI objects by tactile sensations, and the labels of the objects by voice. In order to examine the effectiveness of the proposed method, an evaluation experiment with visually impaired subjects was performed. The results are analyzed, and it is shown that the search process proposed by the authors can find the desired GUI object more quickly with less burden to the user than the search using a keypad. It is also seen that the visually impaired user employs direct operations in the search for GUI objects using tactile sensations. © 1999 Scripta Technica, Electron Comm Jpn Pt 3, 82(8): 40–49, 1999

Demonstration of Lockheed's Computer-Human Interface Rapid Prototyping (CHIRP) Toolkit

The main purpose of this demonstration is to show the capabilities of the CHIRP Toolkit which is being developed at the Lockheed Rapid Prototyping Laboratory in Sunnyvale, CA. CHIRP is designed to aid the human factors engineer in the development, demonstration and evaluation of advanced forms of computer-human interaction (CHI). The demonstration will show the ease with which high fidelity prototypes of interactive graphical user interfaces (GUIs) can be developed using a form of visual programming. CHIRP contains both primitive GUI objects (e.g., a button and a menu selection bar) and higher-level prefabricated interface modules (e.g., a panel of buttons and a complete pull-down menu structure). Reusable maps, globes, images, graphs, analog displays and other useful graphical objects are stored in application graphics libraries. Special interactive graphics creation utilities are provided to support rapid development of application scenarios involving orbital mechanics, image processing, tracking and signal processing. Examples resulting from a variety of rapid prototyping activities involving CHIs for graphics-oriented military and commercial applications will be shown. These examples will illustrate the value of CHI prototyping in all phases of the computer system development cycle. CHIRP-based prototypes have been successfully used for marketing activities, for verification of system requirements, for simulating concepts of operation, for risk reduction studies, for usability evaluations, and for training users. Specific examples shown in this demonstration include a data fusion GUI designed for a proof of concept review, an animated 3D graphics orbital mechanics simulation used for requirements analysis and risk reduction studies, and an interactive GUI remote diagnostics prototype used for workload analysis studies and as a part-task trainer. Since the CHIRP Toolkit is a continuously evolving product, the demonstration will include a brief preview of future developments, including a capability to prototype multimedia CHIs which incorporate imagery, sound, and interactive video.