Digital Library Evaluation Research Articles

A user‐oriented evaluation of digital libraries: case study the “electronic journals” service of the library and information service of the University of Patras, Greece

Provides an evaluation of the use of the electronic journals service of the library and information service of the University of Patras, Greece. Asks who these electronic journals service users are, how often they use the service, what their reasons for use are, where their access points for use are, and which search methods and services they use. In addition, invites users to choose between an electronic and print journal title subscription and indicate some factors that would discourage them from accessing an electronic journals service.

An Evaluation of the Informedia Digital Video Library System at the Open University

The past decade has seen the evolution of physical educational resource collections into digital libraries, a process which has opened up a whole new range of uses beyond the traditional ones of reference and individual study in a fixed location. It is becoming more common for institutions of learning to regard their teaching and learning resource materials as important assets that can be shared, incorporated into new courses and accessed remotely by students. With the inclusion of multiple media (text, video and audio), the potential of digital libraries is further enhanced. Early adopters of this technology are information specialists and authors of learning materials, and in some cases, students. Evaluations of digital libraries that are currently underway at various institutions are aiming to determine whether the technology is usable and cost‐effective. We also see a focus on the kinds of learning activities that can be supported with digital resources and the best ways of organising and describing them. The evaluation described in this paper is primarily concerned with issues of usability and identifying potential areas of use. A planned second phase of the project will delve more deeply into costs and benefits for users in specific contexts of use.

Usability Evaluation of Digital Libraries

Summary This paper distinguishes evaluation procedures in a physical library from those in a digital library, reviewing usability evaluation work within a taxonomy of system design, development and deployment. Usability evaluation of digital libraries should follow two evaluation strategies. The first a convergent methods paradigm. Evaluation data are collected throughout the system life cycle using several different methods. Where comparable interpretations result, evaluators can make confident recommendations for system revision. A second strategy assesses not only the digital library, but the evaluation methods themselves. This self-referring analysis is termed a “double-loop paradigm” and allows evaluators to identify relative efficacy of particular assessment methods for specific library situations. The paper will present evidence for adopting these two strategies, drawing upon the user evaluation effort for the Alexandria Digital Library Project.

Digital Library use in social context

This paper was presented at the 1995 Allerton Conference on Methodologies for the User-centered Design and Evaluation of Digital Libraries and was revised January 1996.

Concept mapping

The Interactive Multimedia Group (IMG) at Cornell University has been designing and researching the impact of collaborative multimedia technologies on educational environments for over ten years. In one of our current projects, the Making of America, the IMG is focusing on the development of tools to support access to digital libraries. This effort is complemented by parallel research into the cognitive, behavioral, and social implications of using these systems. Additionally, the IMG is working in concert with the Museum Education Site Licensing (MESL) project to investigate visual search strategies and the ways in which digital images can be used to enhance classroom instruction and scholarly research.We have found that user acceptance and usability are major issues in the design of digital libraries. In essence, the IMG's research indicates that the design of digital libraries will be most successful when user centered design includes the development and implementation of tools for human-computer interaction and problem-solving.According to Fidel (1993), there is a growing trend in the use of qualitative methods in information retrieval (IR) research. She says that even though the defining characteristics of qualitative research--that it is open and flexible, holistic and case-oriented, inductive and noncontrolling--make its methodologies "best for exploring human behavior in depth, and thus of great relevance to IR research," too few studies have explored their application in-depth (Fidel, 1993, p. 219). The IMG is exploring qualitative research methods that employ rich data to examine complicated computer-mediated environments. These innovative approaches to user- centered design and evaluation respond to the challenges of tomorrow's digital library environments.Beyond the technical design challenges of digital libraries and other information retrieval systems, our studies have demonstrated the need to address the social and psychological aspects of using online resources and collaboration. We have found that parallel development-evaluation approaches provide a wealth of analytical data that can be used to enhance the overall design process.Our approach to building networked environments is to collect descriptive, qualitative data by using sensitive online computer tracking tools combined with innovative ethnographic research methods. We have found that technologically-rich environments demand equally rich data collection, analysis and interpretation tools--ones capable of examining human-computer interactions as well as the social and cognitive dynamics that develop during computer-mediated collaboration.Today, there is little doubt that computer-mediated communication technologies are having a profound impact on how we approach and engage in the processes of education and research. However, our research has demonstrated that students often do not make effective use of available database resources and communication tools (Gay & Lentini, 1995; Gay, 1995; Gay & Grosz-Ngate, 1994; Gay & Mazur, 1993; Trumbull, Gay, & Mazur, 1992). But why?We believe that user interfaces and systems that can accommodate diverse needs and search strategies need to be developed. Likewise, users need to be trained in communication and information seeking behaviors in order to effectively use these powerful systems. We have found that by studying the multiple ways in which users interact with these new systems, we can develop more responsive tools, programs, and technologies. Based on our studies as well as the growing body of research findings from computer-mediated communication research, we have begun the development of a coherent set of principles to guide the design and evaluation of digital libraries.