There are widespread calls for increased demographic diversity in science, often linked to the epistemic claim that including more perspectives will improve the quality of the knowledge produced. By distinguishing between demographic and epistemic diversity, we show that this is only true some of the time. There are cases where increasing demographic diversity will not bring about the necessary epistemic diversity and cases where failing to exclude some voices reduces the quality of the scientific debate. We seek to resolve these tensions with an analysis that turns on the way the experience-based expertise of non-scientists can be absorbed into mainstream science. Mostly it has to be done via what we call 'virtual diversity', in which scientists take responsibility for acquiring interactional expertise in the non-scientific expertise-based domains which they consider provide knowledge valuable to the science. We argue that virtual diversity represents the only feasible option in most scenarios, with cases where demographic diversity or full cultural mergers provide the solution being the exception rather than the rule. This analysis is an exercise in the sociology of knowledge, which is considered as being continuous with philosophy. The paper is prescriptive as well as descriptive, and the moral, cultural, political, and educational implications of the argument are drawn out. A main conclusion is that the acquisition of virtual diversity should be a new norm for science, allowing the voices of experienced non-scientist citizens to be heard but without eroding the institution of science, which continues to be a vital foundation of truth in democracy.