A new statistical model for analyzing rating scale data pertaining to word meaning.

Abstract

The concrete-abstract categorization scheme has guided several research programs. A popular way to classify words into one of these categories is to calculate a word's mean value in a Concreteness or Imageability rating scale. However, this procedure has several limitations. For instance, results can be highly distorted by outliers, ascribe differences among words when none may exist, and neglect rating trends in participants. We suggest using an alternative procedure to analyze rating scale data called median polish analysis (MPA). MPA is tolerant to outliers and accounts for information in multiple dimensions, including trends among participants. MPA performance can be readily evaluated using an effect size measure called analog R 2 and be integrated with bootstrap 95% confidence intervals, which can prevent assigning inexistent differences among words. To compare these analysis procedures, we asked 80 participants to rate a set of nouns and verbs using four different rating scales: Action, Concreteness, Imageability, and Multisensory. We analyzed the data using both two-way and three-way MPA models. We also calculated 95% CIs for the two-way models. Categorizing words with the Action scale revealed a continuum of word meaning for both nouns and verbs. The remaining scales produced dichotomous or stratified results for nouns, and continuous results for verbs. While the sample mean analysis generated continua irrespective of the rating scale, MPA differentiated among dichotomies and continua. We conclude that MPA allowed us to better classify words by discarding outliers, focusing on main trends, and considering the differences in rating criteria among participants.