Sketch2Process: End-to-End BPMN Sketch Recognition Based on Neural Networks

Abstract

Process models play an important role in various software engineering contexts. Among others, they are used to capture business-related requirements and provide the basis for the development of process-oriented applications in low-code/no-code settings. To support modelers in creating, checking, and maintaining process models, dedicated tools are available. While these tools are generally considered as indispensable to capture process models for their later use, the initial version of a process model is often sketched on a whiteboard or a piece of paper. This has been found to have great advantages, especially with respect to communication and collaboration. It, however, also creates the need to subsequently transform the model sketch into a digital counterpart that can be further processed by modeling and analysis tools. Therefore, to automate this task, various so-called sketch recognition approaches have been defined in the past. Yet, these existing approaches are too limited for use in practice, since they, for instance, require sketches to be created on a digital device or do not address the recognition of edges or textual labels. Against this background, we use this paper to introduce Sketch2Process, the first end-to-end sketch recognition approach for process models captured using BPMN. Sketch2Process uses a neural network-based architecture to recognize the shapes, edges, and textual labels of highly expressive process models, covering 25 types of BPMN elements. To train and evaluate our approach, we created a dataset consisting of 704 hand-drawn and manually annotated BPMN models. Our experiments demonstrate that our approach is highly accurate and consistently outperforms the state of the art.