Graphical User Interface Elements Research Articles

WebDraw: A machine learning-driven tool for automatic website prototyping

To overcome the time-consuming nature and improve the cost-effectiveness of classical web development, being automatic is the most convenient alternative recent researchers suggest. Over the years, researchers have been working on inventing new approaches to generating websites automatically. In this paper, a novel approach is presented that automates the website generation process by incorporating web designer best practices and driving new prototype websites without the significant effort of redesigning websites. It takes high-fidelity mock-up design artifacts such as screen captures of real-world websites, and generates functional websites similar to the input websites, which involves three steps: GUI element detection, classification, and code generation. First, image processing techniques are applied to detect atomic web GUI elements from a mock-up design artifact of a real-world website. Second, a Convolutional Neural Network (CNN) is trained to classify the extracted web GUI elements into their domain-specific types such as headings, paragraphs, images, etc. A Graphical User Interface (GUI) is typically represented in code as a hierarchical tree, with nested GUI elements bundled together within one another to construct a tree. A recursive algorithm is proposed that constructs the appropriate Document Object Model (DOM) hierarchy for a website by recursively grouping classified web GUI elements within each other. Finally, the constructed DOM is converted to the accurate native code. The approach was implemented as a tool called WebDraw. Design science research evaluation shows that WebDraw achieves an average of 90% web GUI element classification and generates website prototypes that are visually similar to the target website design mock-up artifact while producing functional GUI code. Furthermore, interviews with industry professionals illustrate WebDraw's industry relevance for improving their industrial web design and development workflows.

Modbus Data Provider for Automation Researcher Using C#

Purpose: Modbus is a popular protocol for data exchange between devices in industrial automation. It has several advantages over other protocols, like noise immunity, long-distance coverage, and easy integration with the microcontroller's serial module. Sometimes we need a device that provides data for our research work. Here we demonstrate a procedure so researchers can create a virtual Modbus client to get Modbus data for their research work. We created a Modbus client in C# language In Visual Studio. We added a couple of modules, Like the Modbus client, serial module, and message display. We added a couple of graphical user interface elements to control the application. The project code is available on GitHub. The researcher can get and customize according to their needs. Design/Methodology/Approach: We created an application in C#. The application has several modules. The main module is the Modbus client. The external device is connected through USB to RS485 converter. When our application starts, the COM object is created. One timer is also started. Its interval is one millisecond. It checks whether the data is reached or not. Once the serial object receives the data, the packet is passed to that Modbus client. The Modbus client starts parsing the received packet. If the packet is OK, then, Extract the command. According to the received command, it created a response packet and added the data of the requested register. Calculate the CRC and add it at the end of the packet. After preparing the packet, send a response back to the master. Findings/Result: Sometimes, the automation researcher does not have the device to provide the research data. Through this research work, we provide a procedure so the researcher can create a virtual Modbus client to provide the data for their research work. So it can be helpful to the researcher to get the data using their working system. We tested it several times with the baud rate of 9600. It is working perfectly without any issues. Researchers can use it for their research work as a software tool. Originality/Value: Several software programs are available to provide the data over the Modbus. Sometimes we need to customize the software according to our requirements. Most of the software is not open source, so here we provide an application so that researchers can optimize and customize the code for their research work so it can provide them with some valuable resources. Paper Type: Experimental-based Research.

GUILGET: GUI Layout GEneration with Transformer

Sketching out Graphical User Interface (GUI) layout is part of the pipeline of designing a GUI and a crucial task for the success of a software application. Arranging all components inside a GUI layout manually is a time-consuming task. In order to assist designers, we developed a method named GUILGET to automatically generate GUI layouts from positional constraints represented as GUI arrangement graphs (GUIAGs). The goal is to support the initial step of GUI design by producing realistic and diverse GUI layouts. The existing image layout generation techniques often cannot incorporate GUI design constraints. Thus, GUILGET needs to adapt existing techniques to generate GUI layouts that obey to constraints specific to GUI designs. GUILGET is based on transformers in order to capture the semantic in relationships between elements from GUI-AG. Moreover, the model learns constraints through the minimization of losses responsible for placing each component inside its parent layout, for not letting components overlap if they are inside the same parent, and for component alignment. Our experiments, which are conducted on the CLAY dataset, reveal that our model has the best understanding of relationships from GUI-AG and has the best performances in most of evaluation metrics. Therefore, our work contributes to improved GUI layout generation by proposing a novel method that effectively accounts for the constraints on GUI elements and paves the road for a more efficient GUI design pipeline.

A graphical user interface for calculating the arterial input function during dynamic positron emission tomography

Purpose. Dynamic positron emission tomography (dPET) requires the acquisition of the arterial input function (AIF), conventionally obtained via invasive arterial blood sampling. To obtain the AIF non-invasively, our group developed and combined two novel solutions consisting of (1) a detector, placed on a patient’s wrist during the PET scans to measure the radiation leaving the wrist and (2) a Geant4-based Monte Carlo simulation software. The simulations require patient-specific wrist geometry. The aim of this study was to develop a graphical user interface (GUI) allowing the user to import 2D ultrasound scans of a patient’s wrist, and measure the wrist features needed to calculate the AIF. Methods. The GUI elements were implemented using Qt5 and VTK-8.2.0. The user imports a patient’s wrist ultrasound scans, measures the radial artery and veins’ surface and depth to model a wrist phantom, then specifies the radioactive source used during the dPET scan. The phantom, the source, and the number of decay events are imported into the Geant4-based Monte Carlo software to run a simulation. In this study, 100 million decays of 18F and 68Ga were simulated in a wrist phantom designed based on an ultrasound scan. The detector’s efficiency was calculated and the results were analyzed using a clinical data processing algorithm developed in a previous study. Results. The detector’s total efficiency decreased by 3.5% for 18F and by 51.7% for 68Ga when using a phantom based on ultrasound scans compared to a generic wrist phantom. Similarly, the data processing algorithm’s accuracy decreased when using the patient-specific phantom, giving errors greater than 1.0% for both radioisotopes. Conclusions. This toolkit enables the user to run Geant4-based Monte Carlo simulations for dPET detector development applications using a patient-specific wrist phantom. Leading to a more precise simulation of the developed detector during dPET and the calculation of a personalized AIF.

VID2META: Complementing Android Programming Screencasts with Code Elements and GUIs

The complexity of software projects and the rapid technological evolution make it such that developers often need additional help and knowledge to tackle their daily tasks. For this purpose, they often refer to online resources, which are easy to access and contain a wealth of information in various formats. Programming screencasts hosted on platforms such as YouTube are one such online resource that has seen a growth in popularity and adoption over the past decade. These screencasts usually have some metadata such as a title, a short description, and a set of tags that should describe what the main concepts captured in the video are. Unfortunately, metadata are often generic and do not contain detailed information about the code showcased in the tutorial, such as the API calls or graphical user interface (GUI) elements employed, which could lead to developers missing useful tutorials. Having a quick overview of the main code elements and GUIs used in a video tutorial can be very helpful for developers looking for code examples involving specific API calls, or looking to design applications with a specific GUI in mind. The aim is to make this information easily available to developers, and propose VID2META, a technique that automatically extracts Java import statements, class names, method information, GUI elements, and GUI screens from videos and makes them available to developers as metadata. VID2META is currently designed to work with Android screencasts. It analyzes video frames using a combination of computer vision, deep learning, optical character recognition, and heuristic-based approaches to identify the needed information in a frame, extract it, and present it to the developer. VID2META has been evaluated in an empirical study on 70 Android programming videos collected from YouTube. The results revealed that VID2META can accurately detect and extract Java and GUI elements from Android programming videos with an average accuracy of 90%.

YOLOv5-MGC: GUI Element Identification for Mobile Applications Based on Improved YOLOv5

The identification of interface elements is the first step in mobile application automated testing and the key to smooth testing. However, existing object detection algorithms have a low accuracy rate, and some tiny elements are missed in the recognition of graphical user interface (GUI) elements. To address this limitation, this paper proposes the YOLOv5-MGC algorithm, a robot vision-based interface element recognition algorithm for mobile applications. The algorithm improves the network by using K-means++ algorithm for target anchor box generation, applying the attention mechanism, adding a microscale detection layer, and introducing the Ghost bottleneck module. The proposed approach enhances the recognition accuracy of the elements through the target anchor box and attention mechanism. Moreover, it enhances the network’s ability to detect tiny elements, which improves the shortcomings of the current target detection algorithm and is conducive to further promoting mobile application robot testing and enhancing robot testing automation. Experimental results show that the YOLOv5-MGC algorithm is superior to the YOLOv5 for object detection in the recognition of GUI elements, with the mean average precision (mAP_0.5) reaching 89.8% and the recognition precision reaching 80.8%.

Gaze Buttons Revisited.Looking for an Effective Way of "Clicking" without the Mouse

The fundamental problem for designing a gaze-based human-computer interaction is related to development of an effective method for activating graphical user interface elements by means of gaze only. Such a method should be easy for the user to apply, however at the same time, it requires eye movements that are clearly different from the natural behavior of the eye. We examined three methods of button activation by gaze, looking for the most effective way of gaze "clicking". These were: 1) the most standard method based on the use of dwell-time, 2) its modification based on detection of fixation located inside the buttons area and 3) and the most novel method based on gaze gestures consisting of movement into the button area and outward movement in the approximately opposite direction. We compared these gaze control methods under homogeneous conditions, which allows for a more reliable assessment of their relative usefulness. Two layouts of buttons were used: arranged on a grid, like on a telephone pad, and on a circle with an empty center. The experimental task was to enter a set of four-digit PINs using a set of gaze buttons corresponding to ten digits. A group of novices were instructed to use all the three methods and both button layouts (six experimental conditions). The activation methods were compared in terms of system usability, objectively measured by the PIN entry speed and the number of errors, as well as using a subjective SUS questionnaire. The system based on gaze gestures was worse in both measures; however, it had its followers. The method based on fixation detection instead of dwell-time did not significantly increase the entry speed due to the greater number of errors caused by non-intentional buttons activation. The circle layout turned out to be generally more convenient than the telephone pad layout.

Assessing the Visual Esthetics of User Interfaces: A Ten-Year Systematic Mapping

ABSTRACT Visual esthetics is recognized as an essential factor of perceived usability, credibility, and overall quality of software systems user interfaces. Thus, it is crucial to assess this factor as part of the development process to improve graphical user interface (GUI) quality. And, although there exist diverse approaches to assess visual esthetics today, they significantly differ in their definition and techniques for measuring it. Therefore, we performed a systematic mapping to provide an overview comparing the existing approaches from the last ten years (2010–2020). We encountered 27 articles reporting different approach types, including assessments measuring user perception of visual esthetics via human responses, metrics based on GUI elements, extracting handcrafted features from GUI screenshots, and deep-learning-based approaches. Most of them implement objective approaches, either using metrics or feature extraction. The majority of the approaches are aimed at web GUIs, with only three focusing on mobile GUIs, considering their smaller screen size, less use of color, and a more vertical element alignment. Few approaches have been comprehensively evaluated regarding reliability and validity. These results indicate the need for further research considering the importance of visual esthetics as part of software quality. The results of this study present an overview and comparison of these approaches from the last decade providing a systematic basis for researchers to better understand the challenges for GUI visual esthetics assessments and the enhancement of these approaches.

Paper2Wire – A Case Study of User-Centred Development of Machine Learning Tools for UX Designers

Abstract This paper reflects on a case study of a user-centred concept development process for a Machine Learning (ML) based design tool, conducted at an industry partner. The resulting concept uses ML to match graphical user interface elements in sketches on paper to their digital counterparts to create consistent wireframes. A user study (N=20) with a working prototype shows that this concept is preferred by designers, compared to the previous manual procedure. Reflecting on our process and findings we discuss lessons learned for developing ML tools that respect practitioners’ needs and practices.

A Kinect-Based Interactive System for Home-Assisted Active Aging.

Virtually every country in the world is facing an unprecedented challenge: society is aging. Assistive technologies are expected to play a key role in promoting healthy lifestyles in the elderly. This paper presents a Kinect-based interactive system for home-assisted healthy aging, which guides, supervises, and corrects older users when they perform scheduled physical exercises. Interactions take place in gamified environments with augmented reality. Many graphical user interface elements and workflows have been designed considering the sensory, physical and technological shortcomings of the elderly, adapting accordingly the interaction methods, graphics, exercises, tolerance margins, physical goals, and scoring criteria. Experiments involved 57 participants aged between 65 and 80 who performed the same physical routine six times during 15 days. After each session, participants completed a usability survey. Results provided significant evidence that support (1) the effectiveness of the system in assisting older users of different age ranges, (2) the accuracy of the system in measuring progress in physical achievement of the elderly, and (3) a progressive acceptance of the system as it was used. As a main conclusion, the experiments verified that despite their poor technological skills, older people can adapt positively to the use of an interactive assistance tool for active aging if they experience clear benefits.

Development of a cyber design modeling declarative Language for cyber physical production systems

This paper considers the issues of software development automation for cyber-physical production systems within the framework of the Smart Manufacturing (SM) and Industry 4.0 concepts. The development of cyber-physical production systems (CPPS) for each enterprise is an individual task that takes into account the specifics of the production process, the requirements for data visualization at each level from Supervisory Control And Data Acquisition (SCADA) to Enterprise Resource Planning System (ERP) in a single information space using different technologies for processing and storing data. To ensure all the requirements that are specified by the customer in the technical requirement (TR), the authors propose to automate the process of developing a cyber-physical production systems Human-Machine Interface (HMI) prototype at an early stage of drafting the terms of reference together with the customer, which will make it possible to take into account all the requirements for visualizations information. Based on the experience gained in the development of cyber-physical production systems, the authors have developed a declarative cyber-design modeling language based on parameters mathematical description of the and Graphical User Interface (GUI) elements events, which allows to simplify the process of HMI cyber-physical production systems development at an early stage by generating fragments of program code for object-oriented programming languages.

VizSciFlow: A Visually Guided Scripting Framework for Supporting Complex Scientific Data Analysis

Scientific workflow management systems such as Galaxy, Taverna and Workspace, have been developed to automate scientific workflow management and are increasingly being used to accelerate the specification, execution, visualization, and monitoring of data-intensive tasks. For example, the popular bioinformatics platform Galaxy is installed on over 168 servers around the world and the social networking space myExperiment shares almost 4,000 Galaxy scientific workflows among its 10,665 members. Most of these systems offer graphical interfaces for composing workflows. However, while graphical languages are considered easier to use, graphical workflow models are more difficult to comprehend and maintain as they become larger and more complex. Text-based languages are considered harder to use but have the potential to provide a clean and concise expression of workflow even for large and complex workflows. A recent study showed that some scientists prefer script/text-based environments to perform complex scientific analysis with workflows. Unfortunately, such environments are unable to meet the needs of scientists who prefer graphical workflows. In order to address the needs of both types of scientists and at the same time to have script-based workflow models because of their underlying benefits, we propose a visually guided workflow modeling framework that combines interactive graphical user interface elements in an integrated development environment with the power of a domain-specific language to compose independently developed and loosely coupled services into workflows. Our domain-specific language provides scientists with a clean, concise, and abstract view of workflow to better support workflow modeling. As a proof of concept, we developed VizSciFlow, a generalized scientific workflow management system that can be customized for use in a variety of scientific domains. As a first use case, we configured and customized VizSciFlow for the bioinformatics domain. We conducted three user studies to assess its usability, expressiveness, efficiency, and flexibility. Results are promising, and in particular, our user studies show that VizSciFlow is more desirable for users to use than either Python or Galaxy for solving complex scientific problems.

Development of measurement instrument for visual qualities of graphical user interface elements (VISQUAL): a test in the context of mobile game icons

Graphical user interfaces are widely common and present in everyday human–computer interaction, dominantly in computers and smartphones. Today, various actions are performed via graphical user interface elements, e.g., windows, menus and icons. An attractive user interface that adapts to user needs and preferences is progressively important as it often allows personalized information processing that facilitates interaction. However, practitioners and scholars have lacked an instrument for measuring user perception of aesthetics within graphical user interface elements to aid in creating successful graphical assets. Therefore, we studied dimensionality of ratings of different perceived aesthetic qualities in GUI elements as the foundation for the measurement instrument. First, we devised a semantic differential scale of 22 adjective pairs by combining prior scattered measures. We then conducted a vignette experiment with random participant (n = 569) assignment to evaluate 4 icons from a total of pre-selected 68 game app icons across 4 categories (concrete, abstract, character and text) using the semantic scales. This resulted in a total of 2276 individual icon evaluations. Through exploratory factor analyses, the observations converged into 5 dimensions of perceived visual quality: Excellence/Inferiority, Graciousness/Harshness, Idleness/Liveliness, Normalness/Bizarreness and Complexity/Simplicity. We then proceeded to conduct confirmatory factor analyses to test the model fit of the 5-factor model with all 22 adjective pairs as well as with an adjusted version of 15 adjective pairs. Overall, this study developed, validated, and consequently presents a measurement instrument for perceptions of visual qualities of graphical user interfaces and/or singular interface elements (VISQUAL) that can be used in multiple ways in several contexts related to visual human-computer interaction, interfaces and their adaption.

Research on EEG-based Graphic User Interface Kansei Design Evaluation

Graphical user interface (GUI) is designed as the interaction medium between the user and the interface, and the perceptual experience of GUI design has been paid more and more attention by users. Based on the theory of perceptual engineering (KE), two groups of different visual style interfaces were taken as an example to record the EEG data when users watched two groups of visual interfaces, in order to explore the user’s perceptual imagery and perceptual experience for the visual interface. It aims to meet the user’s perceptual needs and provide an effective evaluation method and design basis for the graphical user interface design. Firstly, the EEG spectrogram and brain topographic maps were obtained by data analysis and processing. The results showed that the activity levels of the θ wave and α wave induced by the two groups of different visual style interfaces were significantly different. Secondly, this paper analyzed the user’s perceptual imagery with GUI perceptual design elements, and concluded that the perceptual design elements of GUI would affect the user’s cognitive interest and perceptual experience. GUI design should focus on the unity and coordination of perceptual design elements and perceptual imageries. Finally, it is concluded that the EEG-based perceptual design evaluation method can effectively evaluate the GUI visual interface.

A User Interface for Optimizing Radiologist Engagement in Image Data Curation for Artificial Intelligence.

To delineate image data curation needs and describe a locally designed graphical user interface (GUI) to aid radiologists in image annotation for artificial intelligence (AI) applications in medical imaging. GUI components support image analysis toolboxes, picture archiving and communication system integration, third-party applications, processing of scripting languages, and integration of deep learning libraries. For clinical AI applications, GUI components included two-dimensional segmentation and classification; three-dimensional segmentation and quantification; and three-dimensional segmentation, quantification, and classification. To assess radiologist engagement and performance efficiency associated with GUI-related capabilities, image annotation rate (studies per day) and speed (minutes per case) were evaluated in two clinical scenarios of varying complexity: hip fracture detection and coronary atherosclerotic plaque demarcation and stenosis grading. For hip fracture, 1050 radiographs were annotated over 7 days (150 studies per day; median speed: 10 seconds per study [interquartile range, 3-21 seconds per study]). A total of 294 coronary CT angiographic studies with 1843 arteries and branches were annotated for atherosclerotic plaque over 23 days (15.2 studies [80.1 vessels] per day; median speed: 6.08 minutes per study [interquartile range, 2.8-10.6 minutes per study] and 73 seconds per vessel [interquartile range, 20.9-155 seconds per vessel]). GUI-component compatibility with common image analysis tools facilitates radiologist engagement in image data curation, including image annotation, supporting AI application development and evolution for medical imaging. When complemented by other GUI elements, a continuous integrated workflow supporting formation of an agile deep neural network life cycle results.Supplemental material is available for this article.© RSNA, 2019.

3770 Published By: Blue Eyes Intelligence Engineering & Sciences Publication Retrieval Number: B3426078219/19©BEIESP DOI: 10.35940/ijrte.B3426.078219 Journal Website: www.ijrte.org Development of a Simple Graphical Interface Based Software for Machine Learning and Data Visualization

Machine learning has become one of the foremost techniques used for extracting knowledge from large amounts of data. The programming expertise required to implement machine learning algorithms has led to the rise of software products that simplify the process. Many of these systems however, have sacrificed simplicity as they evolved and included more features. In this study, a machine learning software with a simple graphical user interface was developed with a special focus on enhancing usability. The system made use of basic graphical interface elements such as buttons and textboxes. Comparison of the system with other similar open-source tools revealed that the developed system showed an improvement in usability over the other tools.

The Current Status of Accessibility in Mobile Apps

This study evaluated the status of accessibility in mobile apps by investigating the graphical user interface (GUI) structures and conformance to accessibility guidelines of 479 Android apps in 23 business categories from Google Play. An automated tool, IBM Mobile Accessibility Checker (MAC), was used to identify the accessibility issues, which were categorized as a violation (V), potential violation (PV), or warning (W). The results showed 94.8%, 97.5%, and 66.4% of apps studied contained issues related to V, PV, or W, respectively. Five widget categories (TextView, ImageView, View, Button, and ImageButton) were used to create 92% of the total number of the GUI elements and caused 89%, 78%, and 86% of V, PV, and W, respectively. These accessibility issues were mainly caused by lack of element focus, missing element description, low text color contrast, lack of sufficient spacing between elements, and less than minimum sizes of text fonts and elements. Together, these accessibility issues accounted for 97.0%, 77.8%, and 94.5% of V, PV, and W, respectively. This study proposed coverage measures to estimate the percentage of accessibility issues identified by an automated tool. The result showed that MAC, on average, identified about 67% of accessibility issues in mobile apps. Two new accessibility conformance measures were proposed in this study: inaccessible element rate (IAER) and accessibility issue rate (AIR). IAER estimates the percentage of GUI elements that are inaccessible. AIR calculates the percentage of the actual number of accessibility issues relative to the maximum number of accessibility issues. Average IAER and AIR scores were 27.3%, 19.9%, 6.3% and 20.7%, 15.0%, 5.4% for V, PV, and W, respectively, for the studied apps. The IAER score showed approximately 30% of the GUI elements had accessibility issues, and the AIR score indicated that 15% of the accessibility issues remained and need to be fixed to make the apps accessible.

Characterizing mobile apps from a source and test code viewpoint

Context: while the mobile computing market has expanded and become critical, the amount and complexity of mobile apps have also increased. To assure reliability, these apps require software engineering methods, mainly verification, validation, and testing. However, mobile app testing is a challenging activity due to the diversity and limitations found in mobile devices. Thus, it would be interesting to characterize mobile apps in hopes of assisting in the definition of more efficient and effective testing approaches. Objective: this paper aims to identify and quantify the specific characteristics of mobile apps so that testers can draw from this knowledge and tailor software testing activities to mobile apps. We investigate the presence of automated tests, adopted frameworks, external connectivity, graphical user interface (GUI) elements, sensors, and different system configurations. Method: we developed a tool to support the automatic extraction of characteristics from Android apps. We conducted an empirical study with a sample of 663 open source mobile apps. Results: we found that one third of the projects perform automated testing. The frameworks used in these projects can be divided into three groups: unit testing, GUI testing, and mocking. There is a medium correlation between project size and test presence. Specific features of mobile apps (connectivity, GUI, sensors, and multiple configurations) are present in the projects, however, they are fully covered by tests. Conclusion: automated tests are still not developed in a systematic way. Interestingly, measures of app popularity (number of downloads and rating) do not seem to be correlated with the presence of tests. However, the results show a correlation of the project size and more critical domains with the existence of automated tests. Although challenges such as connectivity, sensors, and multiple configurations are present in the examined apps, only one tool has been identified to support the testing of these challenges.

GUICop: Approach and toolset for specification‐based GUI testing

SummaryOracles used for testing graphical user interface (GUI) programmes are required to take into consideration complicating factors such as variations in screen resolution or colour scheme when comparing observed GUI elements with expected GUI elements. Researchers proposed fuzzy comparison rules and computationally expensive image processing techniques to tame the comparison process because otherwise the naïve matching comparison would be too constraining and consequently impractical. Alternatively, this paper proposes GUICop, a novel approach with a supporting toolset that takes (1) a GUI programme and (2) user‐defined GUI specifications characterizing the rendering behaviour of the GUI elements and checks whether the execution traces of the programme satisfy the specifications. GUICop comprises the following: (1) a GUI Specification Language; (2) a Driver; (3) Instrumented GUI Libraries; 4) a Solver; and (5) a Code Weaver. The user defines the specifications of the subject GUI programme using the GUI Specification Language. The Driver traverses the GUI structure of the programme and generates events that drive its execution. The Instrumented GUI Libraries capture the GUI execution trace, ie, information about the positions and visibility of the GUI elements. And the Solver, enabled by code injected by the Code Weaver, checks whether the traces satisfy the specifications. GUICop was successfully evaluated using 4 open source GUI applications that included 8 defects, namely, Jajuk, Gason, JEdit, and TerpPaint.

On the misuse of graphical user interface elements to implement security controls

Abstract GUIs are the predominant means by which users interact with modern programs. GUIs contain a number of common visual elements widgets such as buttons, textfields, and lists, and GUIs typically provide the ability to change attributes on these widgets to control their visibility and behavior. While these attributes are extremely useful to provide visual cues to users to guide them through an application's GUI, they can also be misused for purposes they were not intended. In particular, in the context of GUI-based applications that include multiple privilege levels within the application, GUI element attributes may be misused as a mechanism for enforcing access control policies. This work presents a method to detect misuse of user interface elements to implement access control, it is based on our earlier work C. Mulliner, W. Robertson, and E. Kirda, “Hidden GEMs: Automated Discovery of Access Control Vulnerabilities in Graphical User Interfaces”, in Proceedings of the IEEE Symposium on Security and Privacy, 2014. that introduced the vulnerability class the we refer to as GEMs, or instances of GUI element misuse. Using our GEM detection method we discovered unknown vulnerabilities in several applications.