Evaluating Knowledge of Business Processes

Abstract

1 IntroductionThis paper introduces our research on testing methods that can be employed in testing the knowledge of business processes. This study approaches the research question: Which method of testing the knowledge about processes/procedures is more suited for evaluating the capacity of employees to execute them? Typically, process knowledge is evaluated by asking multiple-choice or open questions on the process documentation. Most organizations document some of the main processes in textual form and then struggle to keep it up to date. A growing number of organizations acknowledge the importance of business processes, document them using some form of graphical models, and employ information systems tailored to support them. However, there are still many organizations that haven't documented processes. In these organizations, employees learn by training and/or experience how to achieve organizational goals. No matter if process documentation exists or not, it is critical for managers to be able to evaluate how well the employees are able to execute processes.Shallow knowledge about processes relates to learning the main steps to be performed, their sequence, the documents and data involved, etc. A deeper understanding resides in, for example, issues like contingency steps in case of errors; overview of organization-wide processes, etc. Current testing based on multiplechoice questionnaires don't go beyond the shallow understanding. We argue that a new testing approach is needed. It should aim to put the tested employee in the position where problem-solving deep knowledge is needed, rather than the ability to memorize process steps. In this paper we introduce a first take on this challenge. We evaluate if asking questions in a graphical -interactive manner is better than the multiple-choice way. Better is interpreted in terms of a greater number of correctly asked questions as well as the time needed to answer comprehension questions. The context of our experiment is an organization of higher education. More specifically, we use the annual admission process to evaluate how accurate different types of participants know and understand the entire process, and if they are able to execute it in any specific case that might arise.The paper unfolds as follows. First, we provide an overview of the theoretical foundations by reviewing papers related to factors influencing model understanding and experimental design. In the next section, we provide the details of our controlled experiment. The single factor of the experiment is the comprehension question presentation format. In section 4 we introduce the results and our data analysis. We end with conclusions and the implications of our findings.2Related WorkBusiness process models are key artefacts in the development of information systems. While one of their main purposes is to facilitate communication among stakeholders, little is known about the factors that influence their comprehension by human agents. To date, the body of research on process model understanding relies on controlled experiments based on multiple-choice comprehension questions. Therefore, in this section, we approach two main related research avenues: one on process model understanding, and a second one on controlled experiments in process model settings.The notation that we use in our experiment is the Business Process Model and Notation (BPMN) which is the industry standard in process modelling. It has straight-forward syntax and semantics. So-called connectors (XOR, AND) define complex routing constraints of splits (multiple outgoing arcs) and joins or merges (multiple ingoing arcs). Through XOR, when splitting, the sequence flows to exactly one of the outgoing branches. When merging, it awaits one incoming branch to complete before triggering the outgoing flow. Through AND, when used to split the sequence flow, all outgoing branches are activated simultaneously. When merging parallel branches process flow waits for all incoming branches to complete before triggering the outgoing flow. …