Assessing the impact of information services in the health sector

Abstract

This special issue of Health Information and Libraries Journal focuses on one of the most critical issues facing health information services and libraries in the 21st century—being able to justify our services in terms of their value to our users and stakeholders. Since the library community first began to seriously engage with performance measurement in the 1970s, we have seen the emphasis change from measuring inputs and processes to investigating outputs, outcomes, service quality and satisfaction, and now impacts. This is more than just a change in terminology; it represents a re-focusing of our attempts to evaluate our services, and reflects a real change in the ethos of our profession. Partly driven by public sector accountability, partly by the threat of new online sources of information that are independent of organized library services, and partly, in the health sector, by the welcome focus on evidence-based practice, we seem to have got the message. This is reflected in the fact that in every issue of HILJ in recent years there is at least one, and usually more than one, paper with words such as evaluating, effectiveness, value, impact or evidence in the title; this is in addition to Andrew Booth's widely appreciated column ‘Using evidence in practice’. This newer term impact sums up a range of dimensions concerning the effectiveness of our services—encompassing as it does the effect of our actions and services on those we serve—our users, the institutions to whose goals we are committed to contributing, and our wider stakeholders, including patients, the public, the government. To know what effect we are having through the services we provide, whether we are achieving our objectives, meeting the needs of our users and enabling them to meet their objectives, and to ensure we provide the most effective services within the financial and other constraints we work under, we need to know how we’re doing, and to be able to measure, in a rigorous objective way, our impacts. These impacts may of course be negative, as well as positive, direct and indirect, may be intended or unintended. Only by investigating them, can we know, report, and act on them, to ensure our services are relevant and valued. Given the number of papers already appearing on this theme why a special issue? Well evaluating your service, and measuring its impacts, although vital, is still a difficult process for many librarians. To know what to evaluate, how to design research that will stand up to scrutiny, what are the most robust (and affordable) methods to use, are ongoing questions. The answers are not static. Evaluation is an evolving process and one in which we can learn from each other by sharing experiences, and developing measurement tools that are both valid and reliable. This issue provides some valuable information and reflection about the processes involved in measuring impacts, the difficulties and rewards that others have experienced, and most importantly the results of these endeavours. There is a wide range of information services that are evaluated in the papers in this issue, and a variety of perspectives on the central topic, from evaluating the impact of services to clinicians in both hospital settings and in the community, the impact of various modes of delivery of information literacy training, and the impact of patient information services. Authors from Canada, Australia, and the USA, as well as the United Kingdom, report impact studies conducted in all these countries, and more, using a wide range of methods, from the rigorously quantitative to the openly ethnographic. They address issues of method, the need for health librarians to have expertise in a range of methods, apply rigorous valid measures, and develop standardised evaluation models so impacts can be compared, and evidence of the effectiveness of services more easily identified. Whatever your service, and whatever your problem, we hope there's something here to contribute to your impact studies. In the lead paper for the issue, ‘Measuring the value and impact of health library and information services: past reflections, future possibilities’ Joanne Marshall reflects on the development of her research focus over the years, and the changing research methods she has used to asses the value and impact of health information services. In particular, Joanne Marshall notes how in the health sector, our familiarity with the research methods of the health sciences have encouraged us to be both innovative and rigorous in our research approaches, and she highlights the continuing need for value and impact studies, and new approaches as we develop new services to meet ever changing user needs. The bibliography prepared for the new Rochester II impact study serves to remind us of the substantial body of knowledge that already exists about the value and impact of health sciences libraries, and the benefits of using both quantitative and qualitative data, to fully address research questions about impacts. Joanne Marshall's research has itself had enormous impact on the field of health information services research, and from her reflections we can take several important messages, about the value of learning from each research project to inform the next, about being eclectic and rigorous in our methods in order to provide evidence of value that will stand up to scrutiny from our stakeholders, and about the value of partnerships in research—between practitioners and academics, librarians and clinicians, in developing research agendas and programmes. Alison Brettle's paper, a systematic review of research ‘Evaluating information skills training in health libraries’ focuses on the need for outcome measures that can demonstrate validity and reliability, concepts which she defines, and then applies to the research she reviews. Without using measures that have been rigorously validated, she demonstrates, we cannot be sure that we are measuring the impact of information literacy training accurately or consistently. Brettle identifies a framework to help practitioners and researchers select a measure: Who is the learner? What is the intervention? What are the outcomes? Does the measure test knowledge or skills (being able to put knowledge into practice)? Her review provides guidance to health librarians by highlighting which measures have been used in various contexts, and identifies those that demonstrate validity and reliability. By replicating studies that have used validated measures, we can build up a body of research that will support meta analysis of findings, and better demonstrate the value and positive impacts of such training. This paper has huge practical applications for those looking to evaluate the teaching they provide. The third paper, by Bruce Thompson, Martha Kyrillidou and Colleen Cook, ‘User library service expectations in health science versus other settings: a LibQUAL study’ takes a very different approach to measuring impacts, and focuses on user perceptions of quality of service received. These perceptions are assessed using LibQUAL+, a methodology for evaluating service quality that has now been used in nearly 1000 libraries in the USA, Canada, Europe, the United Kingdom, Scandinavia, the Middle East, Australia, South Africa and New Zealand. The paper explores the library service expectations and perceptions of users across health-related libraries compared with major research libraries not operating in a medical context, in various countries. A number of health libraries have recently begun to use the LibQUAL+ survey for measuring service quality, and they will no doubt be interested in this analysis of findings across libraries in the health versus other sectors, and perhaps surprised by the findings. Carolyn Medernach and Joanne Franko's paper, ‘Assessing the Impact of Information Services in a Regionalized Health Care Organization’ reports a longitudinal impact study that contrasts usage of medical library services before and after the implementation of several new services, and assesses the clinical impact of the information provided by the medical library. The authors highlight the need for providers of health information services to consider measures that include both the tangible benefits (patient advice, change in diagnosis, change in prescription) and the intangible benefits (increasing knowledge about that patients’ condition, increasing staff satisfaction) of library services, but warn that the use of impact measures may be contingent on the healthcare delivery system in question. The authors discuss the insights they gained from measuring both the use of library services and the impact of using the services i.e. what survey respondents actually did with the information received. ‘Evaluation of outreach services for primary care and mental health; assessing the impact’ by Lyn Robinson and David Bawden is a summary of findings concerning the impact of services, from a larger project carried out for the London Health Libraries. A major intention of this study was to develop some best practice guidelines for outreach information services of this kind in a community setting. Although the authors were able to show some very positive outcomes from the information services delivered, and argue that both justification and improvement of outreach services can legitimately be based on a demonstration of indirect impacts, direct impacts of the kind sought by stakeholders, such as improved patient care, cost savings, etc., were more difficult to establish. Their conclusions, that outreach services of this kind need a more consistent framework to ensure equity of service, and to facilitate standard methods for evaluation, reiterate the message coming through from several papers in this issue—that an ability to look at comparable data from several evaluations will enable us to be clearer about the impact of health information services overall. Services in the domain of patient information are a focus of many health information services, and have been shown in the past to make a positive contribution to patient outcomes. Leesa Bonniface and Lelia Green's paper ‘Finding a new kind of knowledge on the HeartNET website’ reports on patient contributions to a website developed by the National Heart Foundation in Western Australia and Edith Cowan University, and demonstrates a new way in which these positive outcomes can be supported. Using a strongly ethnographic approach, the paper provides a useful insight into the sense of isolation experienced by many heart patients and the value they found in an online community for people with heart conditions. Patients describe how their unsatisfactory search for information on the Internet led them to discover a new ‘place,’ the online community HeartNet, where they could ask difficult questions, and gain support from others. The authors show the very positive impact this kind of social networking (or Web 2.0) application can play in the lives of patients, and argue that directing patients to such resources, along with appropriate cautions, is a legitimate information service for libraries dealing with patient information requests—a message that would apply equally to public as well as health libraries. The final paper in the issue by Pam Bailey and colleagues at the Northumbria University, shows the impact of pro-active study support for groups of students from minority backgrounds entering formal tertiary study after some years of clinical experience. Their paper ‘Assessing the impact of a study skills programme on the academic development of nursing diploma students at Northumbria University, UK’, evaluates an initiative which offered face to face as well as online guidance in study skills to students identified as needing help. Their research shows that these students may lack understanding of what is expected of them as independent learners at the higher level, need additional support, and (a not uncommon finding in the literature) that information literacy education is more effective when it is embedded in the curriculum of such students, and included in their assessment. Their paper shows convincingly the value of assessing the impact of information literacy programmes to provide evidence to justify the costs. There will be many more papers in future issues of Health Information and Libraries Journal that will add to the knowledge reported here, and to this particular collection of methods, approaches and settings, and some that may well develop these impact studies further. We hope in due course, we’ll have a further paper on the Rochester II study that health librarians around the world will be watching. But in the meantime, we should all be encouraged—we are having impact, we are making a difference, and the more we learn about the appropriate tools to measure and improve that impact the greater that impact will become.