Image2emmet: Automatic code generation from web user interface image

Abstract

AbstractWeb development usually follows with analyzing the functionality, designing the user interface (UI) prototype, implementing the UI by front‐end (FE) developers and implementing the REpresentational State Transfer (RESTful) application programming interface (API) by back‐end (BE) programmers. Unfortunately, web development is a tedious, cumbersome, and time‐consuming task, which makes it a challenge for the FE programmers to work in an efficient way. In this paper, we propose an approach, image2emmet, to assist FE programmers in implementing the UI. First, we collect HyperText Markup Language, Cascading Style Sheets (HTML‐CSS) dataset in an automatic and efficient way. The HTML‐CSS dataset used for model training consists of HTML‐CSS code and its display images. Second, the faster region‐based convolutional neural network (CNN) (R‐CNN) is utilized to detect the UI component. Finally, we build a model combining CNN and long short‐term memory (LSTM) to transform the UI component into the HTML‐CSS code. The empirical study demonstrates that image2emmet can achieve a precision of 80% on the UI component detection and 60% on the transformation of UI component into HTML‐CSS code.