A novel syntax-aware automatic graphics code generation with attention-based deep neural network

Abstract

Recent advances in deep learning have made it possible to automatically translate graphical user interface (GUI) into code by an encoder-decoder framework. This framework generally uses deep convolutional neural network (CNN) to extract image features, which are then translated into hundreds of code tokens by a code generator based on a recurrent neural network (RNN). However, there are two challenges in the implementation of this framework: one is how to make full use of the information contained in the GUI and domain specified language (DSL) code, the other is how to make generated DSL code conform to syntax rules. To fully leverage the information in GUI and DSL code, we first propose a model named HGui2Code that integrates visual attention-enabled GUI features (extracted by CNN) with DSL attention-enabled semantic features (extracted by LSTM). Besides, we propose SGui2Code, a novel model that makes use of a ON-LSTM network to generate DSL code that is correct in syntax. HGui2code pays more attention to semantic information, while SGui2code focuses on grammar rules. Extensive experimental results show that our models outperform state-of-the-art methods on the web dataset, yielding 5.5% higher accuracy with the HGui2Code model and 1.5% using the SGui2Code model respectively. Although our models do not have huge boost on IOS and Android dataset, DSL code generated by our models are very close to the layout of components in corresponding GUI.