An approach to improving existing measurement frameworks

Abstract

Software organizations are in need of methods for understanding, structuring, and improving the data they are collecting. This paper discusses an approach for use when a large number of diverse metrics are already being collected by a software organization. The approach combines two methods. One looks at an organization's measurement framework in a top-down fashion and the other looks at it in a bottom-up fashion. The top-down method, based on the goal-question-metric (GQM) paradigm, is used to identify the measurement goals of data users. These goals are then mapped to the metrics being used by the organization, allowing us to: (1) identify which metrics are and are not useful to the organization, and (2) determine whether the goals of data user groups can be satisfied by the data that are being collected by the organization. The bottom-up method is based on a data mining technique called attribute focusing (AF). Our method uses this technique to identify useful information in the data that the data users were not aware of. We describe our experience in analyzing data from a software customer satisfaction survey at IBM to illustrate how the AF technique can be combined with the GQM paradigm to improve measurement and data use inside software organizations.