Graphical User Interface Code Research Articles

WebDraw: A machine learning-driven tool for automatic website prototyping

To overcome the time-consuming nature and improve the cost-effectiveness of classical web development, being automatic is the most convenient alternative recent researchers suggest. Over the years, researchers have been working on inventing new approaches to generating websites automatically. In this paper, a novel approach is presented that automates the website generation process by incorporating web designer best practices and driving new prototype websites without the significant effort of redesigning websites. It takes high-fidelity mock-up design artifacts such as screen captures of real-world websites, and generates functional websites similar to the input websites, which involves three steps: GUI element detection, classification, and code generation. First, image processing techniques are applied to detect atomic web GUI elements from a mock-up design artifact of a real-world website. Second, a Convolutional Neural Network (CNN) is trained to classify the extracted web GUI elements into their domain-specific types such as headings, paragraphs, images, etc. A Graphical User Interface (GUI) is typically represented in code as a hierarchical tree, with nested GUI elements bundled together within one another to construct a tree. A recursive algorithm is proposed that constructs the appropriate Document Object Model (DOM) hierarchy for a website by recursively grouping classified web GUI elements within each other. Finally, the constructed DOM is converted to the accurate native code. The approach was implemented as a tool called WebDraw. Design science research evaluation shows that WebDraw achieves an average of 90% web GUI element classification and generates website prototypes that are visually similar to the target website design mock-up artifact while producing functional GUI code. Furthermore, interviews with industry professionals illustrate WebDraw's industry relevance for improving their industrial web design and development workflows.

A novel syntax-aware automatic graphics code generation with attention-based deep neural network

Recent advances in deep learning have made it possible to automatically translate graphical user interface (GUI) into code by an encoder-decoder framework. This framework generally uses deep convolutional neural network (CNN) to extract image features, which are then translated into hundreds of code tokens by a code generator based on a recurrent neural network (RNN). However, there are two challenges in the implementation of this framework: one is how to make full use of the information contained in the GUI and domain specified language (DSL) code, the other is how to make generated DSL code conform to syntax rules. To fully leverage the information in GUI and DSL code, we first propose a model named HGui2Code that integrates visual attention-enabled GUI features (extracted by CNN) with DSL attention-enabled semantic features (extracted by LSTM). Besides, we propose SGui2Code, a novel model that makes use of a ON-LSTM network to generate DSL code that is correct in syntax. HGui2code pays more attention to semantic information, while SGui2code focuses on grammar rules. Extensive experimental results show that our models outperform state-of-the-art methods on the web dataset, yielding 5.5% higher accuracy with the HGui2Code model and 1.5% using the SGui2Code model respectively. Although our models do not have huge boost on IOS and Android dataset, DSL code generated by our models are very close to the layout of components in corresponding GUI.

An open source graphical user interface for wireless communication and operation of wearable robotic technology.

IntroductionWearable robotic exoskeletons offer the potential to move gait training from the clinic to the community thereby providing greater therapy dosage in more naturalistic settings. To capitalize on this potential, intuitive and robust interfaces are necessary between robotic devices and end users. Such interfaces hold great promise for research if they are also designed to record data from the robot during its use.MethodsWe present the design and validation of an open source graphical user interface (GUI) for wireless operation of and real-time data logging from a pediatric robotic exoskeleton. The GUI was designed for trained users such as an engineer or clinician. A simplified mobile application is also provided to enable exoskeleton operation by an end-user or their caretaker. GUI function was validated during simulated walking with the exoskeleton using a motion capture system.ResultsOur results demonstrate the ability of the GUI to wirelessly operate and save data from exoskeleton sensors with high fidelity comparable to motion capture.ConclusionThe GUI code, available in a public repository with a detailed description and step-by-step tutorial, is configurable to interact with any robotic device operated by a microcontroller and therefore represents a potentially powerful tool for deployment and evaluation of community based robotics.

The jet impingement boiling heat transfer with ad hoc wall thermal boundary conditions

In a two-phase heat exchanger, the thermal boundary condition at the boiling wall plays an import role. To investigate the characteristics of its flow and heat transfer, it is required to solve a three-domain conjugation heat transfer problem which takes into account of boiling, conduction, and air convection. In the current design, the saturated water flows into a cylindrical chamber with a tube array, whereas the hot air travels outside of chamber and boils the water inside. The effects of the water inlet velocity and hot air inlet mass flow rate are measured in the experiments. A simulation tool with Graphical User Interface code is developed for predicting the three-domain conjugation heat transfer. The boiling heat transfer in the complex case is showed to be well explained in the approach of combining the Rensselaer Polytechnic Institute (RPI) boiling model. The experiments indicate that the wall temperature on the solid-air interface and the transferred energy are independent of water inlet velocity but significantly depend on the air inlet mass flow rate. The wall temperatures in the centre core area (tube array region) are relatively uniform, whereas a huge temperature gradient is measured in the peripheral area. The maximum temperature difference in the core region is only around 17.7% or 24.6% of the core-to-peripheral temperature difference in the cases with the high or low air inlet velocity. The experimental observations have been reproduced in the simulation. The heat transfers on the hot air side and the water boiling side significantly influence each other. Prominent variations of wall temperature and heat flux result in a co-existence of single-phase water convection and the water boiling flow. It demonstrates that the conjugation has to be considered with applying an ad hoc thermal boundary conditions in the cases.

Visual GUI testing in practice: challenges, problemsand limitations

In today's software development industry, high-level tests such as Graphical User Interface (GUI) based system and acceptance tests are mostly performed with manual practices that are often costly, tedious and error prone. Test automation has been proposed to solve these problems but most automation techniques approach testing from a lower level of system abstraction. Their suitability for high-level tests has therefore been questioned. High-level test automation techniques such as Record and Replay exist, but studies suggest that these techniques suffer from limitations, e.g. sensitivity to GUI layout or code changes, system implementation dependencies, etc. Visual GUI Testing (VGT) is an emerging technique in industrial practice with perceived higher flexibility and robustness to certain GUI changes than previous high-level (GUI) test automation techniques. The core of VGT is image recognition which is applied to analyze and interact with the bitmap layer of a system's front end. By coupling image recognition with test scripts, VGT tools can emulate end user behavior on almost any GUI-based system, regardless of implementation language, operating system or platform. However, VGT is not without its own challenges, problems and limitations (CPLs) but, like for many other automated test techniques, there is a lack of empirically-based knowledge of these CPLs and how they impact industrial applicability. Crucially, there is also a lack of information on the cost of applying this type of test automation in industry. This manuscript reports an empirical, multi-unit case study performed at two Swedish companies that develop safety-critical software. It studies their transition from manual system test cases into tests automated with VGT. In total, four different test suites that together include more than 300 high-level system test cases were automated for two multi-million lines of code systems. The results show that the transitioned test cases could find defects in the tested systems and that all applicable test cases could be automated. However, during these transition projects a number of hurdles had to be addressed; a total of 58 different CPLs were identified and then categorized into 26 types. We present these CPL types and an analysis of the implications for the transition to and use of VGT in industrial software development practice. In addition, four high-level solutions are presented that were identified during the study, which would address about half of the identified CPLs. Furthermore, collected metrics on cost and return on investment of the VGT transition are reported together with information about the VGT suites' defect finding ability. Nine of the identified defects are reported, 5 of which were unknown to testers with extensive experience from using the manual test suites. The main conclusion from this study is that even though there are many challenges related to the transition and usage of VGT, the technique is still valuable, flexible and considered cost-effective by the industrial practitioners. The presented CPLs also provide decision support in the use and advancement of VGT and potentially other automated testing techniques similar to VGT, e.g. Record and Replay.

A MDA-Based Model-Driven Approach to Generate GUI for Mobile Applications

Developing applications for mobile platforms is a compound task, due to variability of mobile OSs and the number of different devices that need to be supported. Model-Driven Architecture (MDA) approach could provide a possible solution to offer an automated way to generate a Graphical User Interface (GUI) for such applications. In this paper, we propose a MDA-based model-driven approach to generate the GUI for mobile applications. The adopted approach consists of four main steps (i) modeling the GUI under UML; (ii) transforming the obtained diagrams to a simplified XMI schema; (iii) model-to-model transformation; and (iv) model-to-code generation. Our method has the advantages to give a graphical way for designing under UML. Currently, the method has been implemented to support two platforms Android and BlackBerry. The applicability of the approach is demonstrated via a case study that illustrates the GUI code generation for mobile platforms.

WinGXandORTEP for Windows: an update

TheWinGXsuite provides a complete set of programs for the treatment of small-molecule single-crystal diffraction data, from data reduction and processing, structure solution, model refinement and visualization, and metric analysis of molecular geometry and crystal packing, to final report preparation in the form of a CIF. It includes several well known pieces of software and provides a repository for programs when the original authors no longer wish to, or are unable to, maintain them. It also provides menu items to execute external software, such as theSIRandSHELXsuites of programs. The programORTEP for Windowsprovides a graphical user interface (GUI) for the classicORTEPprogram, which is the original software for the illustration of anisotropic displacement ellipsoids. The GUI code provides input capabilities for a wide variety of file formats, and extra functionality such as geometry calculations and ray-traced outputs. The programsWinGXandORTEP for Windowshave been distributed over the internet for about 15 years, and this article describes some of the more modern features of the programs.

Spout: a transparent proxy for safe execution of Java applets

The advent of executable contents such as Java applets exposes World Wide Web (WWW) users to a new class of attacks that were not possible before. Despite an array of security checking, detection, and enforcement mechanisms built into the language model, the compiler, and the run-time system of Java, serious security breach incidents due to implementation bugs still arose repeatedly in the past several years. Without a provably correct implementation of Java's security architecture specification, it is difficult to make any conclusive statements about the security characteristic of current Java virtual machines. The Spout project takes an alternative approach to address Java's security problems. Rather than attempt a provable secure implementation, we aim to confine the damages of malicious Java applets to selective machines, thus, preventing the machines behind an organization's firewall from being attacked by malicious or buggy applets. More concretely, Spout is a distributed Java execution engine that transparently decouples the processing of an incoming applet's application logic from that of graphical user interface (GUI), such that the only part of an applet that is actually running on the requesting user's host is the harmless GUI code. A unique feature of the Spout architecture that does not exist in other similar systems, is that it is completely transparent to and does not require any modifications to WWW browsers or class libraries on the end hosts. This paper describes the detailed design, implementation, and performance measurements of the first Spout prototype, which also incorporates run-time resource monitoring mechanisms to counter denial-of-service attacks.