Detecting and Classifying Problematic Behavior: A Method Based on Multi-Dimensional Software Modeling Behavioral Characteristics

Abstract

Understanding software modelers’ difficulties and evaluating their performance is crucial to Model-Driven Engineering (MDE) education. The software modeling process contains fine-grained information about the modelers’ analysis and thought processes. However, existing research primarily focuses on identifying obvious issues in the software modeling process, such as incorrect connections or misunderstandings, but neglects the behavioral patterns that can reveal underlying, unaddressed modeling problems. This oversight fails to identify deeper problems that do not manifest as obvious issues but still represent significant potential problems in the software modeling process. Our research concentrates on detecting and classifying problematic modeling behaviors from software modeling process data, revealing the potential problems hidden in the process for MDE education. Specifically, we first construct problematic modeling behavior patterns from three dimensions, including anomalous time intervals, repetitions, and frequencies, to further identify characteristics and priorities relevant to problematic modeling behaviors. Then, we design rules with characteristics and priorities to detect and classify problematic modeling behaviors from problematic patterns. To evaluate the effectiveness of our proposal, we apply it to a data-flow diagram modeling platform. This platform can record modelers’ processes and has been practically used in software engineering courses for five years. We have conducted a case study with 12 participants. The macro F1 of detection and classification problematic modeling behaviors is 82.3%, which shows the effectiveness of our approach. Then, to evaluate the usefulness of our proposal for assisting modeling instructors in MDE education, we conducted another case study with 5 modeling instructors. The results show that our approach can help instructors uncover problems hidden in the software modeling process. The results of two case studies demonstrate that our approach effectively detects and classifies problematic modeling behaviors, enabling instructors to better adjust their instructional plans and improve MDE education.