Relationships among Research Design Choices and Psychometric Properties of Rating Scales: A Meta-Analysis

Abstract

The purpose of the article is to propose and empirically test a model of the relationships among research design variables and the psychometric criteria of reliability, convergent validity, discriminant validity, and nomological validity, and among the psychometric criteria themselves. A fundamental premise of the article is that these relationships are much more complex than a casual analysis would suggest. More specifically, though all of the research design variables and psychometric criteria are related at a conceptual level, there is no necessary relationship between many of the variables at an empirical level. Meta-analysis techniques are applied to 162 measures found in the marketing literature to examine the relationships among the research design choices of sampling characteristics, measure characteristics, and measure development processes and the various psychometric criteria of measure quality. The proposed model makes predictions for positive, negative, and no effects relationships. All of the hypotheses in the model except one are supported, with systematic variance apparently the common element producing correlations among the various indicants of measure quality. The primary conclusion is that researchers must pay much greater attention to non-empirical evidence when judging construct validity. Three recommendations are offered for measurement research in marketing.