Mobile GUI Research Articles

Understanding mobile GUI: From pixel-words to screen-sentences

The ubiquity of mobile phones makes mobile GUI understanding an important task. Most previous works in this domain require human-created metadata of screens (e.g. View Hierarchy) during inference, which unfortunately is often not available or reliable enough for GUI understanding. Inspired by the impressive success of Transformers in NLP tasks, targeting for purely vision-based GUI understanding, we extend the concepts of Words/Sentence to Pixel-Words/Screen-Sentence, and propose a mobile GUI understanding architecture: Pixel-Words to Screen-Sentence (PW2SS). In analogy to the individual Words, we define the Pixel-Words as atomic visual components (text and graphic components), which are visually consistent and semantically clear across screenshots of a large variety of design styles. The Pixel-Words extracted from a screenshot are aggregated into Screen-Sentence with a Screen Transformer proposed to model their relations. Since the Pixel-Words are defined as atomic visual components, the ambiguity between their visual appearance and semantics is dramatically reduced. We are able to make use of metadata available in training data to auto-generate high-quality annotations for Pixel-Words. A dataset, RICO-PW, of screenshots with Pixel-Words annotations is built based on the public RICO dataset, which will be released to help to address the lack of high-quality training data in this area. We train a detector to extract Pixel-Words from screenshots on this dataset and achieve metadata-free GUI understanding during inference. We conduct experiments and show that Pixel-Words can be well extracted on RICO-PW and well generalized to a new dataset, P2S-UI, collected by ourselves. The effectiveness of PW2SS is further verified in the GUI understanding tasks including relation prediction, clickability prediction, screen retrieval, and app type classification.

델파이 조사를 통한 모바일 GUI 요소의 운영체제별 컴포넌트 분류

This study is to establish a methodology for systematically conducting comparative research between iOS and Android. A study was conducted to integrally classify different GUI components for each operating system. The research process and results are as follows. First, data on mobile design elements were collected, and detailed elements of mobile GUI components were organized by operating system. Second, after designating five mobile design elements as representative factors, Delphi method analysis was conducted to classify 22 types of GUI components for each mobile operating system into detailed elements. Through the survey results, items that meet the Content Validity Ratio(CVR) were selected. The basis was to conduct research until an agreement was reached.. As a result, since Coefficient of Variation(CV) was secured in the second round, the final Delphi results were summarized. Third, according to the analysis, iOS was classified into four bars, two Navigation, five View, nine Control, and two Button, while Android was classified into three Bar, two Navigation, seven View, eight Control, and two Button. These research methods and results can be used as basic data to build an integrated methodology comparing mobile operating systems.

"니팅머신의 사용 편의성을 위한 모바일 GUI 디자인 사용성 평가원칙 개발 연구: 커스터마이징 앱을 중심으로"

"니팅머신의 사용 편의성을 위한 모바일 GUI 디자인 사용성 평가원칙 개발 연구: 커스터마이징 앱을 중심으로"

Translation from layout-based to visual android test scripts: An empirical evaluation

Mobile GUI tests can be classified as layout-based – i.e. using GUI properties as locators – or Visual – i.e. using widgets’ screen captures as locators –. Visual test scripts require significant maintenance efforts to be kept aligned with the tested application as it evolves or it is ported to different devices.This work aims to conceptualize a translation-based approach to automatically derive Visual tests from existing layout-based counterparts or repair them when graphical changes occur, and to develop a tool that implements and validates the approach.We present TOGGLE, a tool that translates Espresso layout-based tests for Android apps to Visual tests that conform to either SikuliX, EyeAutomate, or a combination of the two tools’ syntax. An experiment is conducted to measure the precision of the translation approach, which is evaluated on maintenance tasks triggered by graphical changes due to device diversity.Our results demonstrate the feasibility of a translation-based approach, show that script portability to different devices is improved (from 32% to 93%), and indicate that translation can repair up to 90% of Visual locators in failing tests.GUI test translation mitigates challenges with Visual tests like maintenance effort and portability, enabling their wider use in industrial practice.

섬진강 하구역 Mobile MGIS 구축 연구

Abstract The study of Mobile MGIS (Marine Geographic Information System) based on the Android Mobile Platform is mainly performed on effective methodologies which transform real world data to computing world data. Mobile GUI system has its own target on reliable data service by acquisition of geometric information using EFDC (Environmental Fluid Dynamics Code) Model, accurate measurement and graphical visualization. Even simple visualizations can aid in the interpretation of complex hydrodynamic relationships that are frequently encountered in the marine environment.The Mobile MGIS provides an easy way for hydrodynamics geoscientists to construct complex visualizations that can be viewed with free software. This study proposes a Mobile GUI MGIS using EFDC Model result of Seomjin River estuary. Finally, we design a Seomjin River estuarine Coastal Waters Mobile MGIS integrating above data models. It must adds more ecological information and the various service item for approach more easily in order to user.Key words : Marine Geographic Information System, Graphic User Information, Android, Mobile Platform, Seomjin River, Environmental Fluid Dynamics Code ModelCorresponding author : 061-659-7155, kimjk@chonnam.ac.kr( ( )(2 )).

Declarative GUI descriptions for device-independent applications

The increasing number and diversity of devices connected to the Internet open new research challenges in the field of cross-platform and device-independent applications. One of the approaches to this problem is the Device-Independent Architecture, which provides application logic and application data device independence. It enables also usage of user interface (UI) adaptation middleware to support application UI device independence. Potentially, device-independent descriptions of application UI can be implemented with existing user interface description languages (UIDLs). In this paper, we present an analysis of eight popular UIDLs that are assumed to be suitable for device-independent GUI descriptions, along with a summary of evaluation results and lessons learned. The selected UIDLs were employed to describe a set of GUI views based on an existing mobile application. The gathered results confirm our research hypothesis that the analyzed popular declarative UIDLs are not capable of describing mobile GUI in a device-independent manner. Therefore, using the knowledge gathered from the reported experiment, we propose a set of guidelines for an optimal device-independent UIDL.

Application of Mobile GUI Design Theory to the Development of an Open Source Touchscreen Smartphone GUI

It can be argued that the focal point of every smartphone is its GUI. The Graphical User Interface is the aspect of any touchscreen smartphone with which users directly interact. Of late, the touchscreen smartphone has become ubiquitous. As such there is a myriad of smartphone platforms, each of which boasts a user-friendly GUI in its own accord. This paper outlines the design and development of an open source touchscreen smartphone GUI, implemented on an Embedded Linux development board, which serves as a hypothetical smartphone. It gives an overview of GUI design theory and mobile usability studies and shows the application of this theory in the programming process for an embedded platform. This paper goes in depth into the development process for an ARM-based Linux platofrm. The major outcomes of this study include the successful formulation and design of a hierarchical, touchscreen GUI suitable for a smartphone, as well as successful development and target-specific implementation of this GUI on an Embedded Linux, ARM-based platform, by adopting of an open source philosophy.

MobiGUITAR: Automated Model-Based Testing of Mobile Apps

As mobile devices become increasingly smarter and more powerful, so too must the engineering of their software. User-interface-driven system testing of these devices is gaining popularity, with each vendor releasing some automation tool. However, these tools are inappropriate for amateur programmers, an increasing portion of app developers. MobiGUITAR (Mobile GUI Testing Framework) provides automated GUI-driven testing of Android apps. It's based on observation, extraction, and abstraction of GUI widgets' run-time state. The abstraction is a scalable state machine model that, together with test coverage criteria, provides a way to automatically generate test cases. When applied to four open-source Android apps, MobiGUITAR automatically generated and executed 7,711 test cases and reported 10 new bugs. Some bugs were Android-specific, stemming from the event- and activity-driven nature of Android.