Evaluating quality of answers in consortial chat reference

The latest issue of College and Research Libraries (January 2009, volume 70, number 1) has an interesting article by Deborah L. Meert and Lisa M. Given ("Measuring Quality in Chat Reference Consortia: A Comparative Analysis of Responses to Users' Queries") reporting on a study that attempted to measure the quality of answers provided in a consortial chat reference service to one member library's patrons. The final version of the article isn't available online yet but you can read the preprint here (PDF). 

The library where the chat service was studied is at the University of Alberta, which until a few years ago used 24/7 Reference and then QuestionPoint to provide a chat reference service available around the clock (they now use LiveHelp and staff the service themselves for limited hours each week). By looking closely at chat reference transcripts from sessions where University of Alberta patrons were helped, the authors of the article wanted to compare the quality of answers given by University of Alberta staff to the answers given by staff of other libraries in the chat reference consortia (i.e., non-University of Alberta staff).

The basic methodology of the authors was as follows:

• select a representative pool of chat transcripts from the University of Alberta service
• code each chat session as either answered or not answered
• code the question type in each session (library user information; request for instruction; request for academic information; miscellaneous or non-library)
• code each chat session as being handled by either U of A staff or non-U of A staff
• for the subset of chat sessions that were coded as "not answered," code to indicate the reason why the questions weren't answered in real time (some were adequately answered after email followup with the patron)

I'll pass over detailing how the chat transcripts were selected and representative samples found. Instead, I want to offer some comments on the undiscussed historical context of the service studied, the sample set used, the coding methodology, the findings, and the implications. The authors say that they used transcripts from October 1 to April 30, but they don't mention what years that covers; they do say that the sessions used are from the first year of service but no launch dates for the service are provided. It would be useful to know exactly when the sessions took place, as there are some historical events that might help place the findings in notable context. I am pretty sure that the chat sessions took place when the University of Alberta was just using 24/7 Reference for its software; my college, too, was also a subscriber at the time, and I know that the number of other libraries in the academic consortial service grew rapidly. For the purposes of this article, it would be helpful for readers to know more about the consortia that was helping to provide answers to University of Albert patrons; specifically, how many other libraries were members of the consortia during the time period that the transcripts came from. How busy was the consortial service at the time? (The busier the service, the less likely that consortial librarians would be willing to expend a lot of time with any one patron and the more likely they would be to refer a question for e-mail follow up by the patron's home library).

Another notable detail is that during the time period that the Unversity of Alberta was in the 24/7 Reference academic cooperative service, they were the only library not in the United States. In their analysis of why non-answered questions weren't actually answered, the authors mention one possible reason as being a "cultural barrier" between librarian and patron, such as "not understanding the Canadian educational context" (80). It seems to me that the authors should have noted more explicity the reality that all the librarians from other libraries in the consortia worked at American colleges and universities and most if not all of the backup librarians employed by 24/7 Reference were also American.

Given that the University of Alberta, then, was the only Canadian institution in the consortia, I wonder just how representative these chat sessions are. I wish I could recall how many academic libraries were in 24/7 Reference when the University of Alberta was; if I had to guess, I would say at least fifty, maybe one hundred.

I did not see any mention of inter-rater reliability in the article. A note one way or the other ("yes, we did it" or "no, we didn't") would have helped me assess the validity of the coding of transcripts.

Another question that came to mind as I read this was whether assessing "answers" is the best way to evaluate a service. It is worth remembering that the study could have looked at how users perceived the quality of the service (do you feel like your question was answered? would you use the service again? would you recommend the service to someone? etc.)  or how they acted in response to the service (how many repeat users were there). I'm not sure I know the answer, but it is a question to keep in mind as you read this article.

The authors found that U of A staff met the library's own reference standards 94% of time; non- U of A staff 82% of the time. They also discovered that  U of A staff answered 89% of the questions in real-time (no followup or referrals were needed), while non-U of A did so only 69% of the time. Meert and Given say that for the consortial service to work at the U of A, the library there must:

provide adquate and easily accessible information to non-UofA staff (assuming that non-UofA staff use this information) that allowed them to answer most questions regarding library user information correctly, and in real time, this would decrease the number of questions not meeting the UofA reference management standards and would increase the number of questions anwered in real time by non-UofA staff. (82)

They go to say that this study "should provide further assurance that high standards of quality can be achieved by nonlocal staff in a chat reference consortium" (83) as long as "consortium staff...have the information they need to answer the most commonly asked types of questions, particularly the kind described in the 'Library User' question category" (82).

It's a shame that the University of Alberta no longer uses a consortial service, because I think it would really great to do this study again. Since the university stopped being in the consortia, it has undergone a number of notable changes. First, 24/7 Reference got acquired by OCLC and merged into its already existing QuestionPoint service. Second, QuestionPoint then built an all new chat software based on the 24/7 Reference software. Third, the consortia has grown considerably since the days that the University of Alberta was in it. I would love to know if service is better or worse now.

A factor that isn't addressed in this article is that while the staffing of chat reference at the University of Alberta may have been relatively stable, the consortia itself was constantly adding new libraries. With each new library in the consortia comes a bunch of librarians brand new to consortial chat reference; it takes quite a bit of experience to be able to provide reference in the consortial environment. Also, experienced staff already in the consortia may retire or take other jobs. It can be quite a challenge to make sure that everybody monitoring a consortial chat service is well-prepared to help patrons a libraries flung across the continent.

I must say that despite my comments, I think this is a good article and well-worth reading. As the authors point out, there has not been much work done yet in looking at how well patrons are helped in consortial chat reference services. This article offers an interesting methodology that I hope others will replicate.