GUI Elements Research Articles

GUI Behaviors to Minimize Pointing-Based Interaction Interferences

Pointing-based interaction interferences are situations wherein GUI elements appear, disappear, or change shortly before being selected, and too late for the user to inhibit their movement. Their cause lays in the design of most GUIs, for which any user event on an interactive element unquestionably reflects the user’s intention—even one millisecond after that element has changed. Previous work indicate that interferences can cause frustration and sometimes severe consequences. This article investigates new default behaviors for GUI elements that aim to prevent the occurrences of interferences or to mitigate their consequences. We present a design space of the advantages and technical requirements of these behaviors and demonstrate in a controlled study how simple rules can reduce the occurrences of so-called “ Pop-up -style” interferences and user frustration. We then discuss their application to various forms of interaction interferences. We conclude by addressing the feasibility and trade-offs of implementing these behaviors in existing systems.

Curio: A Dataflow-Based Framework for Collaborative Urban Visual Analytics.

Over the past decade, several urban visual analytics systems and tools have been proposed to tackle a host of challenges faced by cities, in areas as diverse as transportation, weather, and real estate. Many of these tools have been designed through collaborations with urban experts, aiming to distill intricate urban analysis workflows into interactive visualizations and interfaces. However, the design, implementation, and practical use of these tools still rely on siloed approaches, resulting in bespoke systems that are difficult to reproduce and extend. At the design level, these tools undervalue rich data workflows from urban experts, typically treating them only as data providers and evaluators. At the implementation level, they lack interoperability with other technical frameworks. At the practical use level, they tend to be narrowly focused on specific fields, inadvertently creating barriers to cross-domain collaboration. To address these gaps, we present Curio, a framework for collaborative urban visual analytics. Curio uses a dataflow model with multiple abstraction levels (code, grammar, GUI elements) to facilitate collaboration across the design and implementation of visual analytics components. The framework allows experts to intertwine data preprocessing, management, and visualization stages while tracking the provenance of code and visualizations. In collaboration with urban experts, we evaluate Curio through a diverse set of usage scenarios targeting urban accessibility, urban microclimate, and sunlight access. These scenarios use different types of data and domain methodologies to illustrate Curio's flexibility in tackling pressing societal challenges. Curio is available at urbantk.org/curio.

CRUD Operation on WordPress Database Using C# And REST API

Purpose: IoT is famous for research with sensor data. The standard way is to keep sensor data inside IoT platforms like AWS, Google, and Microsoft Azure cloud. We must pay to keep or exchange the data among different systems. Here, we provide a different approach to keep our research data inside the cloud. We created a communication channel using the C# application and WordPress website database over REST API, which is now widespread. Our data format is JSON, which is the industry standard. Upload data to the website database using an HTTP client from the C# application. The uploaded data can be accessible from anywhere among the various systems. Our website runs round the clock, so our data is always available. This procedure might be cheaper than the available solution due to the use of our existing website database. Through the practical approach, we demonstrate how to do that. This procedure can be helpful for the researcher trying to keep their research data at a cheap cost inside the cloud. The complete project code is available to download, and the value addition to this work. Design/Methodology/Approach: We first install “Local” software inside our system. On that, we installed a WordPress website. Then, we create a custom route for our experiment to differentiate from existing website content inside the server using PHP. Then, we added our module, which provides CURD operation. This module receives JSON-structured content, which is converted by the WordPress engine. The module parses the command and processes it accordingly. The response is returned to the client application. We created a C# application with a couple of GUI elements to interact with the user. We added a couple of modules using the Nuget package manager for REST API. After the application opens, the first action is to connect with the database. Once the database is available, CRUD operation can be executed. Findings/Result: We tested the entire project using practical deployment. The performance is pretty good. A small quantity of network latency is present in our observation. The content updation time depends on website bandwidth, network traffic, the system’s specification, and the number of running applications. Originality/Value: Over the net, there is some practical documentation on C# REST API on the WordPress Website. Sometimes, researchers working on WordPress websites want to use database operation over REST API. Here, we demonstrate this through practical examples. So, it is a slightly different approach for quickly understanding the communication flow. Paper Type: Experimental-based Research.

Semantic Similarity-Based Mobile Application Isomorphic Graphical User Interface Identification

Applying robots to mobile application testing is an emerging approach to automated black-box testing. The key to supporting automated robot testing is the efficient modeling of GUI elements. Since the application under testing often contains a large number of similar GUIs, the GUI model obtained often contains many redundant nodes. This causes the state space explosion of GUI models which has a serious effect on the efficiency of GUI testing. Hence, how to accurately identify isomorphic GUIs and construct quasi-concise GUI models are key challenges faced today. We thus propose a semantic similarity-based approach to identifying isomorphic GUIs for mobile applications. Using this approach, the information of GUI elements is first identified by deep learning network models, then, the GUI structure model feature vector and the semantic model feature vector are extracted and finally merged to generate a GUI embedding vector with semantic information. Finally, the isomorphic GUIs are identified by cosine similarity. Then, three experiments are conducted to verify the generalizability and effectiveness of the method. The experiments demonstrate that the proposed method can accurately identify isomorphic GUIs and shows high compatibility in terms of cross-platform and cross-device applications.

Retracted: Visual Identification of Mobile App GUI Elements for Automated Robotic Testing.

[This retracts the article DOI: 10.1155/2022/4471455.].

모바일 기기 UI 디자인의 방향 연구 ╶ 안드로이드의 미니멀 앱 아이콘 디자인을 중심으로╶

Despite the recent development of displays, people's attention is focused on the importance of UI design, and various studies have been conducted, it is difficult to find studies related to the problems of excessive minimalism that currently appear everywhere. Therefore, this study starts with the question of the present time when the design between the app GUI elements lacks differentiation, each individuality disappears, and the boundaries between the elements become ambiguous.
 Minimalist UI design has become one of the most important design systems of modern times. Previous scuremorphism designs consumed more pixels than needed for real-life metaphor representation at limited display sizes. This has become a very fatal drawback in small displays such as mobile devices, and it can be seen that a change to a simple design that consumes fewer pixels is inevitable as mobile devices become popular. However, these minimalist designs are being abused and misused, reducing user usability and intuition, and making individuality between each element disappear.
 Therefore, in this study, we will look at the design trends and reasons for changes in the UI so far and derive problems with the current UI design, away from ambiguity in the future. After that, we seek the elements that UI design must have to escape from excessive minimalism. Finally, I would like to study the direction in which each element can have differentiation and move toward various design directions.

Virtual IoT Device in C# WPF Using Sinric Pro

Purpose: IoT is becoming the leading player in the industrial automation environment. In most scenarios, we experiment with IoT using a physical device. We can also research using a virtual device that can perform as real hardware. Without buying any physical hardware, we can visualize the status of the operating load or device which is being triggered through the IoT server or client endpoint. The researcher who is not from an electronics background or does not have sufficient knowledge to continue IoT research can do experiments using it. This paper will show how we can create such kinds of virtual instruments or devices. We use the C# client application adopted from Sinric Pro. We simplified the client module for new researchers in the IoT field for easy understanding. We added some Graphical user elements to display the status of the devices in real-life we see. The code is available for customization. Design/Methodology/Approach: We create and configure the device inside the Sinric Pro IoT server. Download the C# client and customize it for better understanding. We added some graphical elements to display the status of the load. Then we trigger the load from any endpoint like Alexa, the server dashboard, or a mobile phone application. Findings/Result: We get better load status visibility using the GUI element and a minimalistic code structure to send or receive the data to and from the IoT server. Through this concept, we made IoT development or demonstration easy. With slight modifications, we can use this procedure to communicate with any IoT server. Originality/Value: Various clients of fake load for IoT are available. Here we are experimenting more realistic way. Fetching the status and triggers the load using visual indication as the real world does. So it will be more understandable to us, and also, we can trigger the load from our GUI, which has the feature to send the value to the respective load. Paper Type: Experimental-based Research.

A Smart IDE for Robotics Research

Purpose: We see the robot is engaged in every task, from the hazardous industrial environment to the floor cleaning. We are benefited directly or indirectly from robots in our everyday life. This is the untired efforts of many Robot researchers who play a significant role behind the scenes. They pass lots of tests before releasing the robot for humans. Most of the test procedure is to send the command from the IDE to the robot. The researcher generally feels two types of constraints. One is various IDE. They spend lots of time finding the tool's location for different robot vendors with their IDE. Another one is, sometimes the proprietary robot IDE is not too fit for robot researchers. They need some customizable IDE. But in the close source application is not possible to customization. Here we described a procedure so that our two constraints can be solved. We developed a smart robotic IDE that can operate various robots, introducing a hardware abstraction layer. This IDE can also be configured in every aspect. It is an open-source application. The robot researcher can easily customize it according to their need. Design/Methodology/Approach: We create a GUI application in C# inside the visual studio community edition 2022. The main UI thread is integrated with file handling, programming interface, and various robot object. Here we created a couple of objects. In this way, we can make any new robot object. The object is exposed on the GUI element. Selecting any robot, we can send the available command through the terminal or programming interface. Findings/Result: Sometimes, robot researchers must break some impenetrable barrier for their research needs. But need lots of effort to develop some custom application to interact with robot or automated device. Through this research work, we provide helpful information so they can integrate their robot easily. Compared to other available open-source or closed-source applications, this application will take less time to debug and solve the issue at the earliest. Originality/Value: The smart IDE makes research work better and faster. We need complete freedom to interact with robots or automated devices. The industry standard robots have their IDE to interact with their robot. Sometimes times more flexibility makes work easy. In this scenario, our approach can provide us with better flexibility to work. By following this procedure, the researcher can get some unique benefits for their research work. Paper Type: Simulation-based Research.

Visual Identification of Mobile App GUI Elements for Automated Robotic Testing.

Automated robotic testing is an emerging testing approach for mobile apps that can afford complete black-box testing. Compared with other automated testing approaches, automatic robotic testing can reduce the dependence on the internal information of apps. However, capturing GUI element information accurately and effectively from a black-box perspective is a critical issue in robotic testing. This study introduces object detection technology to achieve the visual identification of mobile app GUI elements. First, we consider the requirements of test implementation, the feasibility of visual identification, and the external image features of GUI comprehensively to complete the reasonable classification of GUI elements. Subsequently, we constructed and optimized an object detection dataset for the mobile app GUI. Finally, we implement the identification of GUI elements based on the YOLOv3 model and evaluate the effectiveness of the results. This work can serve as the basis for vision-driven robotic testing for mobile apps and presents a universal approach that is not restricted by platforms to identify mobile app GUI elements.

DEVELOPMENT OF A SOFTWARE MODULE FOR OPERATIONAL DISPATCH CONTROL OF PRODUCTION BASED ON CYBER-PHYSICAL CONTROL SYSTEMS

The subject of research in the article is production management processes based on cyber-physical systems. The purpose of the work is to automate the management of production processes using the cybernetic component and analyze the results. To achieve this goal, it is necessary to solve the following tasks: develop an analytical and logical model of the CPPS development management process taking into account the requirements specified in the terms of reference for the development CPPS; to present the analytical and logical structure of connections in the form of a model of interaction of the main windows and all the necessary graphical elements for the implementation of the full-featured interface of the developed CPPS; to carry out experimental researches of efficiency and practical approbation of the received theoretical results, by comparison of classical methods and the developed ones. Results: The targets, physical and cybernetic components of the CPPS development management process were combined in a single information space from the beginning of development to its implementation. The process of checking the achievement of the main goal of CPPS development has been automated, which makes it possible to make changes and manage the CPPS development process at any level and stage of the proposed technology. The process of managing the development of the cybernetic component is automated on the basis of synthesized algorithms of functioning using the GUI elements of object-oriented programming languages, which made it possible to increase the flexibility of the proposed architecture and technologies for automating the CPPS development process and to reduce the development time of the cybernetic component. Conclusions: the proposed models and methods will not only automate the process of managing the development of CPPS from scratch, but will also make it possible to automate the process of managing the development of the cybernetic component for the modernization and improvement of existing CPPS.

FUNCTIONAL TEST AUTOMATION SYSTEM FOR UNITY APPLICATIONS

The tool for automatically executing and validating test sets of actions is not native to applications, which causes problems with recognizing interaction elements due to the abundance or, conversely, lack of GUI elements. This work describes the design and development of an internal tool that provides accurate recognition of interaction elements by decorating the input reading mechanisms of the Unity game engine. Previously, there were no solutions for Unity to automate success tests, testing the main flow of a use case, or other approaches for functional testing of applications, except through the mechanisms of the standard unit testing package. However, no matter how good the approach of using unit testing tools is, in order to cover all possible scenarios of actions within the project, you would have to either write your own test module for each of the aspects, or implement a complex universal system on its basis. The article's graceful workaround for the technical limitation of the standard Input class dramatically reduces the time required for the application testing phase. In addition to solving testing problems, such a form of recording user actions in a virtual environment can be used to generate training simulator scenarios, which will help to avoid the routine work of generating training tracks. So, immersion in a virtual biotechnology laboratory of an expert who can correctly pass all phases of the selected analysis (for example, ELISA) is enough to record the checkpoints of such a scenario for teaching students.

Study on the Usability and Satisfaction of Mobile Banking GUI Elements for Chinese Students

한국 교육부가 2019년 유학생 통계서에 따라 2017년부터 한국에 온 중국유학생들은 점점 많아진다. 또한, 중국 현재 모바일 결제와 모바일 뱅킹 사용은 생활화되고 사용성의 욕구가 점점 높아지는 기대감을 보일 수 있다. 그러나 한중 모바일 뱅킹의 차이로 시각 조형성의 이질감과 불편함을 유발할 수 있다. 이에 국내 대학 입학 확대 추세의 선행수치 결과에 따라 중국유학생이 국내 모바일 뱅킹 GUI 시각적 요소에 따른 사용성과 만족도에 관한 연구가 필요하다 사료되었다. 본 연구는 한국과 중국 대표적 모바일 뱅킹을 분석한 후 중국 모바일 뱅킹의 GUI 요소와 한국 주요 은행이 중국어 환경으로 리뉴얼 된 모바일뱅크 GUI 사용성 및 만족도는 차이를 Interview& Surveys 연구 방법으로 검증하였다. 그 결과 중국유학생들은 한중 대표적인 모바일 뱅킹 중에서 중국건설은행과 KB스타뱅킹의 사용성은 높게 점수를 알 수 있었고 NH올원뱅킹은 낮은 점수를 보이며 상대적으로 미비할 수도 있다는 것을 알 수 있었다. 그리고 중국유학생들은 주요 인터넷 결제와 이체로 이용하고 수많은 정보와 기능을 한눈에 파악할 수 있게 디자인되어야 하고 기능의 아이콘은 일관성 있게 제공하는 것이 좋고 만족도에 긍정적인 효과를 발휘할 수 있을 것이다.

Recent Progress in Automated Code Generation from GUI Images Using Machine Learning Techniques

The manual transformation of a user interface design into code is a costly and time-consuming process. A solution can be the automation of the generation of code based on sketches or GUI design images. Recently, Machine Learning approaches have shown promising results in detecting GUI elements for such automation. Thus, to provide an overview of existing approaches, we performed a systematic mapping study. As a result, we identified and compared 20 approaches, that demonstrate good performance results being considered useful. These results can be used by researchers and practitioners in order to improve the efficiency of the GUI design process as well as continue to evolve and improve approaches for its support.

Development of measurement instrument for visual qualities of graphical user interface elements (VISQUAL): a test in the context of mobile game icons

Graphical user interfaces are widely common and present in everyday human–computer interaction, dominantly in computers and smartphones. Today, various actions are performed via graphical user interface elements, e.g., windows, menus and icons. An attractive user interface that adapts to user needs and preferences is progressively important as it often allows personalized information processing that facilitates interaction. However, practitioners and scholars have lacked an instrument for measuring user perception of aesthetics within graphical user interface elements to aid in creating successful graphical assets. Therefore, we studied dimensionality of ratings of different perceived aesthetic qualities in GUI elements as the foundation for the measurement instrument. First, we devised a semantic differential scale of 22 adjective pairs by combining prior scattered measures. We then conducted a vignette experiment with random participant (n = 569) assignment to evaluate 4 icons from a total of pre-selected 68 game app icons across 4 categories (concrete, abstract, character and text) using the semantic scales. This resulted in a total of 2276 individual icon evaluations. Through exploratory factor analyses, the observations converged into 5 dimensions of perceived visual quality: Excellence/Inferiority, Graciousness/Harshness, Idleness/Liveliness, Normalness/Bizarreness and Complexity/Simplicity. We then proceeded to conduct confirmatory factor analyses to test the model fit of the 5-factor model with all 22 adjective pairs as well as with an adjusted version of 15 adjective pairs. Overall, this study developed, validated, and consequently presents a measurement instrument for perceptions of visual qualities of graphical user interfaces and/or singular interface elements (VISQUAL) that can be used in multiple ways in several contexts related to visual human-computer interaction, interfaces and their adaption.

자전거 헬스케어 앱 GUI 요소별 사용성 분석

현대인들은 각종 동호회 및 취미생활을 즐기거나 스포츠 활동의 참여를 통하여 건강한 삶을 누리기를 원하고 있다. 또한, 스마트폰 발달로 인해 유비쿼터스(Ubiquitous)환경이 보편화 되면서 운동 관리 및 건강 사후 관리 서비스가 가능한 다양한 헬스케어 앱도 출시되고 있다. 이는 개인 맞춤형 서비스뿐 만 아니라 공유 기능을 통해 다양한 커뮤니케이션을 가능하게 만들고 더 나아가 인간중심의 U-헬스케어 패러다임까지 변화시키고 있다. 여가 스포츠 중에서도 4대강을 중심으로 자전거 도로 건설이 진행되면서 자전거 이용자는 해마다 증가 추세에 있다. 이로 인해 자전거 이용자는 수많은 정보를 빨리 얻길 원하고 있으며, 나에게 맞는 운동 강도조절 등의 적극적인 데이터를 얻기를 원한다. 하지만 국내에서 제작된 헬스케어 자전 거앱은 오픈라이더 외에 전문적으로 자신의 운동강도를 체크하고 자전거에 필요한 여러 데이터를 얻는데 한계가 많다. 따라서 본 연구는 국내외 자전거 헬스케어 앱의 GUI에 따른 사용성을 분석하고 국내 자전거 서비스 앱의 개선방향을 제시하고자 한다. 이를 위해 자전거 헬스케어 앱 중 구글 스토어에서 인기가 높은 5가지 앱을 통해 이를 분석하였고 GUI에서도 구분적 특성, 위계적 특성, 보조적 특성으로 나누어 사용성을 평가하였다. 그 결과, 위계적 요소가 사용성에 가장 많은 영향 주고 있었으며, 그 다음으로 구분적 요소 마지막으로는 보조적 특성이 영향을 주고 있는 것으로 나타났다. 따라서 헬스케어 자전거 앱은 이미지나 일러스트레이션들의 다양한 콘텐츠나 버튼등의 구분적, 보조적 특성보다는 색상이나 명도, 채도 차의 위계적 특성을 가장 많이 고려하여 자전거 헬스케어앱을 제작해야 할 것이다.

Inventions on User Configurable GUI A TRIZ Based Analysis

The graphic user interface is a revolutionary development over the prior art technique of the command line interface. Using GUI the user can do even complex operations through simple means like clicking buttons or dragging various graphic elements. Ideally the GUI should contain all those buttons (or menu items or other GUI elements) and only those buttons (or menu items or other GUI elements) that the user needs. Besides the GUI elements should not obscure the display of data, should take no time for searching and should create no confusion in any level of user. Generally the GUI elements are all pre-coded and built into the system by the developer. Even the experienced developer cannot realistically anticipate what options will be required by the users at a future point of time. As it is ultimately the user who knows his requirement, it is better that the user himself configures the GUI according to his requirement. The article discusses the problems involved in implementing user-configurability and illustrates twelve inventions from US Patent database on the same issue. Each invention is analyzed from a TRIZ perspective to illustrate the contradictions, Ideal Final Result and Inventive Principles.

Inventions on Improving Visibility of GUI Elements - A TRIZ Based Analysis

The evolution of computing technology has witnessed tremendous growth in the processing power of the computer. The memory and storage capabilities are also increasing year after year. In contrast to the above, the display area of a PC monitor remains the same. This inhibits the productivity of a computer, as the user does not just have enough view area to interact with the computer.Ideally the display screen should accommodate all required data, objects and GUI elements for user interaction. If some data or object is hidden behind or remaining beyond the display area, it should automatically come to the front when needed by the user. If any text or image is reduced in size to save display space, it should automatically be enlarged when the user wants to view it. The article analyzes six patents from US Patent database illustrating their background, their objectives or desired results, contradictions faced in achieving the desired results and how these inventions have solved the contradictions.

Using TRIZ for Improving Navigation in Graphical User Interface

A typical user interface consists of several buttons, menus, windows, trees and other type of controls. The increased number of GUI elements and complexities of the GUI controls necessitate the user to acquire certain level of skill and efficiency in order to operate the GUI. There are many situations which further make the navigation difficult. Ideally the user should face no difficulties in navigating through the user interface. Any operation in the graphical user interface should require minimum pointer operations from the user (Ideal Final Result).This objective of a graphical user interface has led to several inventions trying to make the navigation simple and efficient. The article analyzes some inventions from US Patent database and illustrates how each of them has tried to solve different problems of navigation in a graphical user interface.

Factors Affecting Cosmetic Mobile App Performance and the Moderating Effect of User Motivation

Purpose: This study aims to analyze the effect of GUI elements on the functional elements of cosmetic brand apps, the effect of functional factors on their usability and usefulness, and the moderating effect of user motivation. Methods: In this study, four elements (launcher icon, layout and menu, typography, and color) were derived as GUI components of the branded app. It will then analyze the effects of these GUI components on the functional elements of mobile apps such as flexibility, efficiency, aesthetics, and ease of use. Regression analysis and multivariate ANOVA were then performed using SPSS v.23. Results: The four aforementioned GUI elements have a positive impact on the flexibility, efficiency, simplicity, and aesthetics of branded apps. Simplicity and aesthetics positively affect the usability of the apps, and flexibility and efficiency also positively affect usability. Analysis results on the moderating effect of user motivation found that it had a significant influence only on the usefulness of the brand app. Conclusion: According to the research results, when developing cosmetic mobile apps, it is necessary to appropriately combine functional elements and GUI elements that affect the performance index in accordance with the users’ usage motivation.

A Study on GUI Elements of a Domestic Cosmetic Apps for Case Analysis: Focusing on Amorepacific and LG Household & Health Care Mobile Brand Apps

A Study on GUI Elements of a Domestic Cosmetic Apps for Case Analysis: Focusing on Amorepacific and LG Household & Health Care Mobile Brand Apps