What is wrong with individual differences research?

Abstract

There are major but largely-unrecognised problems with all individual differences research which uses questionnaires or tests. Firstly, measurement specialists are vocal in observing that rating scales and ability scales cannot, in principle, measure individual differences. This may make psychometrics “a pathology of science” (Michell, 2000). Secondly, online data collection is known to produce seriously flawed results unless great care is taken when designing studies as inattention, “clicking for cash”, and automated systems (“bots”) masquerading as people have been shown to seriously affect the quality of online data. Thirdly, some scales have been developed through repeatedly paraphrasing the same item. We discuss why such scales are problematical, and why they are often validated inappropriately. Finally, we show that scales which supposedly measure quite different traits contain items having almost identical meanings. This artificially inflates the correlation between these scales and will inevitably lead to meaningless analyses, “models” and interpretation of correlations. We discuss some possible solutions to each of these issues.