Can we trust measures of trust? a comparison of results from open and closed questions

Abstract

Many public opinion surveys compare trust in a number of different information and (mediated) knowledge sources, typically using closed questions with a set of answer categories that are imposed by researchers. We aim to validate these categories by quantitatively comparing survey responses about trustworthy sources using open and closed questions, and by qualitatively analyzing the open answers. The results show that answer options typically used for closed questions in academic research are generally valid and closely match categories that respondents come up with unprimed. In some cases, answers to open questions can be non-exhaustive, particularly when sources are considered trustworthy but are not salient for respondents. Open questions, however, may still be useful for exploratory research or more detailed investigations of media diets on the outlet-level. Qualitative approaches to open questions can also give more insight into motivations for distrust, e.g. perceptions of inconsistency or a fundamental rejection of the shared factual basis of an issue. In addition, our results indicate that respondents’ interpretation of answer categories may change reported levels of trust: those that think of more specific outlets tend to report higher general media trust. This study provides new insights into how question design, and particularly the choice of answer options, may influence reported levels and sources of trust, and how qualitative and quantitative approaches to trust measurement can be combined.