Assessing data analysis performance in research contexts: An experiment on accuracy, efficiency, productivity and researchers’ satisfaction

Abstract

Any knowledge generation process involves raw data comprehension, evaluation and inferential reasoning. These practices, common to different disciplines, are known as data analysis, and represent the most important set of activities in research contexts. Researchers use data analysis software methods and tools for generating new knowledge in their daily data analysis. In recent years, data analysis software has been incorporating explicit references in modelling of cognitive processes, in order to improve the assistance offered in data analysis tasks. However, data analysis software commercial suites are still resisting this inclusion, and there is little empirical work done in knowing more about how cognitive aspects inclusion in software helps researchers in analyzing data.In this paper, we evaluate the impact produced by the explicit inclusion of cognitive processes in the assistance logic of software tools design and development. We conducted an empirical experiment comparing data analysis performance using traditional software versus data analysis performance using software-assistance tools which incorporate cognitive processes in their design. The experiment is designed in terms of accuracy, efficiency, productivity and user satisfaction during the data analysis made by researchers. It allowed us to find some clear benefits of the cognitive inclusion in the software designed for research contexts, with statistically significant differences in terms of accuracy, productivity and researcher's satisfaction in support of this explicit inclusion, although some efficiency weaknesses are detected. We also discuss the implications of these results for the priority of cognitive inclusion in the software tools design for research contexts data analysis.