ABOUT THIS SITE: Usability Testing

Anatomy of a (More) Usable Design:
UCSD Libraries Include Audience Perspective In Web Site Development Process

By: Laura Smart, Amy Butros, Esme Cowles, Janet Tait,
Brian Tingle, Esteban Valdez, and Brad Westbrook

Abstract: The UCSD Libraries began its ongoing Portal Project in April 1999. The Portal is intended to integrate the 14 existing branch library sites into a single site with a consistent presentation, designed to meet our users' needs. This case study discusses how user-centered design was incorporated into the development process, the results of our usability testing, and the associated improvements brought to bear on the UCSD Libraries web site.

Contents:
The Development Environment
Planning for Usability
Paper and Pencil Tests
Incorporating Changes
Online Prototype Testing
Incorporating Changes
Beta Release: Iterative Testing Fixes and Challenges
Steady State: Ongoing Usability Assessment
References

The Development Environment

Return to Contents

The branches and specialized collections of the UCSD Libraries are geographically distributed on campus. Our web presence reflected this dispersal. It was heterogeneous. Each branch library maintained its own web site. Common resources and services were redundantly sustained. Duplication of effort cost valuable staff time. Our experience indicated that our users had a difficult time distinguishing between the physical and organizational divisions of our libraries and that our web organizational scheme obscured their access to resources. We wanted to develop a cohesive web site which met the needs of our constituents.

In April 1999 the UCSD Libraries launched the Portal Project. Portal is a large-scale endeavor involving a steering committee of 10 people representing various departments within the campus library system. Steering committee developed the goals and scope of the project, determined the target audience, listed the project requirements and outlined tasks for working groups.

Target Audience

The Portal will improve access to UCSD Libraries Resources and other selective resources. In its first phase the Portal will concentrate on serving the following primary clientele:

The various working groups met weekly and developed a first draft of the portal based upon the project requirements. The user evaluation/testing group began meeting in October 1999 to ensure that our new site met the needs of our primary clientele.

We defined usability simply by equating it with functionality. If the user succeeded in his or her task then our site was useful. We did not address issues of convenience or reliability due to the time constraints of the development project. The first step was to address the details of what we wanted to test and why. We brainstormed a list of features from the first prototype of portal which required analysis. This list evolved into problem statements and test objectives.

Define novice vs. advanced users
Discover users' meaningful subject label grouping/hierarchy
Develop home page labels with "scent." (1)

We outlined our development deadlines and decided which types of usability activities to perform. Usability is a generic term applied to several types of inspection and design methods. The portal project was in an early concept development phase. We needed to select a user-centered design technique appropriate for this project stage and for our problem statements and test objectives. We discussed card sort, heuristic walk though, focus groups, log file analysis, paper and pencil tests, prototype testing, task analysis and formal usability lab methods. Detailed descriptions of usability methods are available from Rubin, Ehrlich and Rohn, and Nielsen. (2)

Our first priority was determining how to label the various links on the home page and how to group them in ways meaningful to the user. Paper and pencil testing was the best fit for our needs.

In paper and pencil testing users are, "shown an aspect of a product on paper and asked questions about it, or asked to respond in other ways. The questions can range from particular attributes, such as organization and layout, to where one might find certain options or types of information."(3) The users are required to make choices or match options regarding their probable choices or interpretations while using the service.(4)

We developed a terminology exercise to discover the terms our audience would use to describe library services and resources. We developed a list of 29 possible terms based on: the labels used on the first draft prototype of the portal interface; a review of terminology used on other academic library home pages; and a brainstorming session for simpler alternatives. We also developed a list of library tasks based on the most frequently asked questions at the libraries' reference desks. We asked participants to match their favorite label to each task on the paper and pencil test.

We identified undergraduate students, graduate students, faculty, administrative staff, and library staff as classes of test subjects that we wanted to target. We developed a demographic questionnaire so we could categorize our results and determine the influence, if any, of level of experience on the home page label terminology. We defined anybody who recognized the name of our online library catalog as an experienced user. All others were novices. We also created an orientation script to be read by the facilitator at each test. Standardized instructions could reduce bias by ensuring that all testers received the same directions for performing the test.

We sampled our test on several naïve users before administering it to our target audience. Pre-testing allowed us to refine our questions and eliminate ambiguities. For example, our initial task was: Find the book, "The Wind in the Willows." Pre-testing indicated that users were confused as to whether they were to find the book at UCSD, or anywhere in the world. Pre-testing also helped us refine the layout. We initially had all of the label choices on two pages and it was easier for the testers if all of their options were available on one single page.

We seized the opportunity to test staff and students by setting up a table outside of the annual staff association pancake breakfast and setting up a table in our student center during lunchtime. We rewarded participants with $5 debit cards to be used for the purchase of food or sundries at various campus cafeterias and retail outlets. We were able to administer the test to 10 staff, 10 undergraduates and 8 graduate students. We attempted to get faculty participation by inviting our personal contacts. Response was nil, perhaps because we requested involvement via campus mail.

We found that there were some differences between the labels preferred by novice and advanced users. We also found some differences between users first and second preferred label. We combined first and second choices for an aggregate preference and created labels which reflected that.

For example, on the task: "Find the book Jurassic Park by Michael Crichton," 35% of respondents selected our online catalog "ROGER" as their first or second choice label. 31% of respondents selected "Find Books and more…" as either their first or second choice of label. Our final label on the second prototype was "ROGER" with a subheading "UCSD's library catalog for books, journals and more."

We used the new labels on our initial prototype interface, which we code named "Portlet." Further testing would determine if they were successful.

The portal project moved into the product design phase. A graphic designer was hired to create a "look and feel" for the second interface prototype. We reviewed our problem statements. The paper and pencil tests helped us ensure that novice users would select the most appropriate link to complete a given library task. The first tests also demonstrated how novice and experienced users differed in their preferred labels. We still needed to answer questions 3-5:

Prototype assessment testing was the most appropriate methodology for the next stage of the development cycle. This type of testing explores how well the end user can actually perform real-life tasks using the prototype. Users are asked to perform the tasks while "talking aloud" about the choices they make.

We reviewed the common library tasks we had used for the terminology test. We rewrote the questions and re-tested them with colleagues to ensure that they were easy to understand. We modified the wording based on that feedback and narrowed the number of tasks down from sixteen to eight. We created a data log sheet to make note taking easier for observers and to help ensure consistency between different observers.

In March 2000 we tested ten library staff volunteers. None of the usability testing team had prior experience administering prototype tests. We began with our colleagues to gain experience and practice before testing with our primary audience of students and faculty. We interviewed each volunteer after testing. During this debriefing participants provided specific details regarding interface difficulties they encountered with the prototype.

The first prototype test indicated several areas for improvement within the interface. Most of our initial testers thought that the interface was easy to use. Their familiarity with the research tools as library staff could have biased their responses. We felt that this bias was acceptable for the purpose of gaining test administration experience.

1. Most testers had difficulty locating journal articles
2. Most testers were unsure what they were searching with the search box on the main page
3. Most users who used the search box got lost navigating the results output page
4. There was no clear path that testers selected when attempting to find library classes -- the "scent" was distributed across the site
5. Testers got stuck where content was under development

1. Make article search tools more prominent from main page, clarifying wording
2. Add explanation to our search box
3. Improve results output from our web spider (time to load, moving table of contents)
4. Add library classes link to home page quick links and to all other relevant library services pages
5. Complete development of unwritten content

We spent the rest of the spring incorporating changes and preparing the web site for beta release in July 2000. The new database driven site reduces redundancy since one web site can be dynamically reproduced on various subject web pages. It simplifies maintenance for web authors. Much of the pre-beta preparation involved populating the database and completing the writing of static services pages such as hours and frequently asked questions.

The beta version of the libraries web site was released July 1, 2000. The new site included Sage our database of recommended scholarly information sources selected by subject specialist librarians. We had not named our database during the initial prototyping. We had labeled it simply "Web Resources and more," based upon our paper and pencil tests. We hoped that naming the database would help users identify what they were searching.

We did not entirely replace our old site with the new graphical look and feel. Instead we provided a link to the new site so our customers could preview the changes (termed a "soft" roll out). We wanted to continue prototype assessment testing before our official launch. In our initial prototype testing we only made improvements after observing several people using the site. Studies have shown that an iterative testing cycle can be a more efficient and cost-effective way to discover usability problems (5) We developed a weekly testing sequence and planned on rapid fixes between each test where possible. The succession allowed us discover more of the initial usability issues and gave us time to find new additional problems within our fixes.

We solicited volunteers from our library lobby and via word-of-mouth with our campus colleagues. We provided gifts to our testers in appreciation of their time (candy filled university mugs and movie passes). We were unable to implement weekly testing in June and July due to production delays. We were able to do limited testing and improvements during those months. We began eight weeks of iterative testing in August 2000.

1. Confusion regarding Sage -- users were still not sure what they were searching from the home page
2. Continued difficulty with journal article searching

1. Draft a "How to Find Articles" help page - VIEW July 2000 version (GIF format)
2. Add explanatory text to Sage link on home page.

We tested again in July 2000 after incorporating these changes using the same instruments and questions as the June tests.

1. Tester easily located "How to Find Articles" page, but still did not succeed in finding an article. He scrolled up and down the page and returned to home without making an appropriate journal database selection.

We removed blocks of text and simplified a number of options available on "How to Find Articles" since our July tester only scanned the help instructions available on that page.

Between July and August testing we reexamined our prototype testing tasks and changed them slightly to reflect what we learned. We also continued to modify our test instrument. Our testers had expressed frustration with their continued struggle to locate journal articles and image files. We wanted to ensure that testers were not having difficulty due to their affective state during the test. We thought we could increase the security and confidence of our testers by providing the easier questions first. We did not remove the difficult questions entirely because they demonstrated areas where the interface needed attention. We instead moved the problematic questions to the end of the test.

We made numerous changes during this phase of development. The following are examples of the types of changes that we made.

2. Added explanatory details to e-journal browsing pages so that users could easily jump to other recommended resources in that subject area.

3. Changed the Browse Sage by Subject pages by adding an explanation of Sage and removed advanced search options. Users were still confused about what Sage meant. Using the advanced features resulted in zero hits in some cases because in some subject areas did not have resources in some format types.

4. Completely revised the How Do I.... page. Users continually chose this option from quick links on the home page whenever they got lost or were confused about what link to take. The original page was based upon our frequently asked questions. It's title "FAQ" as misleading since the label on the home page said "How Do I..." Also, the FAQ page had many internal anchor links which led to navigation confusion. The new "How Do I..." page matches the home page label and is simpler to navigate. The new page contains a large amount of new content and is still evolving.

Some usability problems may not have been identified in our testing to date, such as problems with more advanced features or problems performing tasks that were not on our tests. By the same token, some usability problems may go away as users learn the site and get help from librarians or their peers. Also, some other data that will give us clues about the usability of the site will come from other sources, such as site usage statistics and informal conversations with users. To track these changes as time goes on, we plan to periodically do usability testing on an ongoing basis.

1. Peter Pirolli of Xerox PARC coined the usage of the term "informavores" to describe people hunting for information and noted that users seem to follow the scent of information.

Source: Spool, J., Klee, M., and Schroeder, W. Designing for Scent. Bradford, MA: User Interface Engineering, 2000. Part of Designing Information-Rich Web Sites series. Available for purchase at www.UIEreports.com

2. Rubin, Jeffrey. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. New York: John Wiley & Sons, 1994. p.19-23.

Elrich, K. and Rohn, J. "Cost Justification of Usability Engineering: A Vendor's Perspective" in Cost Justifying Usability. Bias, R. and Mayhew, D. (eds.)Boston: Academic Press, 1994. p.83-87.

Nielsen, J. Usability Inspection Methods. New York: John Wiley & Sons, 1994. p.5-6

4. Boling, E. and Spiaggia, C. Glossary of Usability Activities. Handout from presentation.

5. Nielsen, J. "Guerilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier" in Cost Justifying Usability. Bias, R. and Mayhew, D. (eds.)Boston: Academic Press, 1994. p. 245-272

Anatomy of a (More) Usable Design: UCSD Libraries Include Audience Perspective In Web Site Development Process

Anatomy of a (More) Usable Design:
UCSD Libraries Include Audience Perspective In Web Site Development Process