Automatic Extraction of Design Decisions from Issue Management Systems: A Machine Learning Based Approach

Abstract

The need to explicitly document design decisions has been emphasized both in research and in industry. To address design concerns, software architects and developers implicitly capture design decisions in tools such as issue management systems. These design decisions are not explicitly labeled and are not integrated with the architecture knowledge management tools. Automatically extracting design decisions will aid architectural knowledge management tools to learn from the past decisions and to guide architects while making decisions in similar context. In this paper, we propose a two-phase supervised machine learning based approach to first, automatically detect design decisions from issues and second, to automatically classify the identified design decisions into different decision categories. We have manually analyzed and labeled more than 1,500 issues from two large open source repositories and have used this dataset for generating the machine learning models. We have made the dataset publicly available that will serve as a starting point for researchers to further reference and investigate the design decision detection and classification problem. Our evaluation shows that by using linear support vector machines, we can detect design decisions with 91.29% accuracy and classify them with an accuracy of 82.79%. This provides a quantitative basis for learning from past design decisions to support stakeholders in making better and informed design decisions.