The Critical Assessment of Research: Traditional and New Methods of Evaluation

Abstract

This book is not a guide to evaluating and interpreting clinical studies in the tradition of Richard K. Riegelman's Studying a Study and Testing a Test (Wolters Kluwer/Lippincott Williams & Wilkins; 2013; ISBN: 978-0-78177-4-260). Rather, it explores the contexts in which research is conducted and disseminated. As the authors point out in the introductory chapter, we are continuously bombarded with information derived from research in fields as varied as finance, health care, and climatology. None of us are experts in all of these fields, but we need to decide how to invest our savings, what medicines to take, and whom to elect to represent us in public office. How can we assess this information and make the best possible decisions? Scholars and librarians use a set of traditional tools to evaluate report publications or base their conclusions on research. These tools include author credentials, publisher reputation, and, of course, peer review. Alan Bailin and Ann Grafstein review these “gold standards” and acknowledge their value. However, they also point out that economic and ideological considerations can limit the effectiveness of these widely accepted measures of the reliability of research results. Using real-life examples, the authors explore how these considerations can impede the conduct of research and determine how (or even if) results are reported and disseminated. Chapter 3 focuses on the inhibiting effects that vested interests can have on research and the application of results. Bailin and Grafstein explain how pharmaceutical industry sponsorship led to the promotion of hormone replacement therapy despite the adverse effects of the drugs and describe how the conflict of interest between Arthur Andersen's auditing and consulting services contributed to the collapse of Enron. Chapter 4 discusses how prevailing paradigms can determine the course that research takes and even whether research questions are posed at all. Here, too, the illustrative cases include a medical example, such as the slow abandonment of the idea that peptic ulcers are caused by stress despite the research showing that Helicobacter pylori infection is the primary cause. Chapter 5 discusses the dissemination of research results, again arguing that ideology, paradigms, and economic interests can significantly influence what gets published and what is either not published or is relegated to less traditional venues. Having done their best to cultivate healthy skepticism in their readers, Bailin and Grafstein turn in the last chapter to advice. They review a variety of tools—including newspapers and magazines, Web 2.0 resources such as blogs and wikis, and scholarly publications—that readers can use to become more savvy consumers of research results. Readers may wonder whether this book, which was published nearly four years ago, is still relevant. It is. Human nature, which is at the root of the problems described here, does not change quickly. It is reasonable to expect that cases like the ones cited in this book will happen again. Bailin and Grafstein prescribe no panacea, really, except to point out that naively accepting research results without evaluating them in their economic, political, and social contexts is dangerous. Seasoned librarians will not learn a great deal from this book, but it was not written solely, or even primarily, for librarians. Still, it has a few pearls, such as tips on searching the “gray” literature. And librarians are as loath to change their paradigms as anyone, so the insights offered by Bailin and Grafstein are worth while. The book includes an index and several pages of references, presumably chosen with especially skeptical care. The Critical Assessment of Research is apparently now sold only as an e-book by the publisher, but print copies may still be available from bookstores or online sellers.