The general questions in evaluations are: Why evaluate? What to evaluate? How to evaluate? This chapter is focused on the how approach and specifically on the research methods proposed in order to carry out a qualitative evaluation of digital libraries. Qualitative analysis is associated with empirical information about the world in the form of words. This valuable data can be used to better understand any phenomenon. Or they can also be used to gain new perspectives on issues about which much is already known, or to gain more in-depth data that may be difficult to convey quantitatively. Also, qualitative methods are appropriate in situations where one needs to first identify the variables that might later be tested quantitatively, or where the researcher has determined that quantitative measures cannot adequately describe or interpret a situation.
In terms of digital libraries the goal of qualitative research is to identify and define users' perceptions, opinions and feelings about digital libraries. Users are invited to express their opinions on a number of issues related to digital libraries and researchers to probe and explore their views. These issues might deal with the usefulness, usability or performance of digital libraries.
The most prevailing forms of collecting data about the Information Search Process (ISP) of users are: interviews, observations and focus groups. Information Search Process refers to users' cognitive and operational aspects before, during and after searching a digital library. Specifically, analysing users' perceptions, opinions and feelings during every searching stage can provide valuable qualitative data, such as information about usability issues, about the quality of information provided to users or about the performance of the system regarding recall, precision, relevance and response time. Most recent studies have shown user-logging data as an important method of qualitative evaluation. The great advantages of transaction logs are not simply their size and reach, but also the fact that they are direct and immediately available records of what users have actually done and not what they say they might or would do. But, most importantly user-logging data can be used for much more than generating usage statistics. For example, information about users' navigation choices could be used to inform decisions about page design and layout. Or error rates and user actions to recover from errors may provide useful information about the skill level of typical users, and this information might influence future decisions about interface design. Or determining the structure of relations among documents retrieved from users and analysing these relationships might give valuable data about the usefulness of digital libraries or the structure of DL user community.
Another source of information that can be invaluable to qualitative researchers is the analysis of existing documents, such as personal diaries, emails or transcripts of conversations that describe users' impressions and reactions regarding digital libraries.