A content analysis process for qualitative software engineering research

Abstract

A review of the qualitative research methods discussed in papers that study software engineering teams showed most of those papers did not follow a systematic process during the qualitative analysis. This finding is concerning as this deficiency in research analysis procedure may reduce the validity and/or completeness of the qualitative results. Such a lack of rigor may be a result of qualitative research not being as firmly established in software engineering as quantitative research methodologies. In engineering research, quantitative methods are typically more prevalent and qualitative analysis is part of a mixed-method analysis process. However, when researching teams, where human activity is abundant, qualitative analysis may need to take precedence. In this paper, we focus on the qualitative analysis method called content analysis with the goal of presenting a rigorous process for content analysis in the context of software engineering. We then present and demonstrate the use of that content analysis process for software engineering researchers using two examples. An analysis of 215 articles that were a result of a mapping study on software engineering team research is presented. Those papers were analyzed to determine which utilized a qualitative data analysis method in their research in addition to the rigor and type of qualitative analysis performed. We ultimately included 23 papers in this study. We present a mapping study and a content analysis process that include a straightforward way to select, code, and present data in both inductive and deductive studies. We demonstrated the process using the keywords from the papers included in this study as well as on a second data set that utilized responses from structured interview transcripts from practicing software engineers. The first dataset also resulted in a taxonomy to categorize software engineering team research. We presented and demonstrated a content analysis process in terms of software engineering in order to improve future qualitative software engineering research that would benefit from systematic content analysis.