Tag Archives: Usability

Academic Papers Blog expert-review information-architecture information design iphone maps readability site-navigation usability testing user-task-analysis user experience design user interface design video Writing

Formal usability testing with eye tracking – Mealographer

Academic Papers, Projects, Usability, Writing, , , , , , , ,

Usability Testing

Usability tests can be seen to fall into two general categories, based on their aim: tests which aim to find usability problems with a specific site, and tests which aim to prove or disprove a hypothesis. This test would fall into the former category. A search of the literature will reveal that tests looking to uncover specific usability problems often use a very small number of participants, coming from Nielsen’s (2000) conclusion that five users is enough to find 85 percent of all usability problems. Nielsen derived this formula from earlier work (Nielsen and Landauer, 1993). Although there is much disagreement (Spool and Schroeder, 2001), this rule of thumb has the advantage of fitting the time and money budget of many projects.

Use of Eye-Tracking Data

In terms of raw data, eye tracking produces an embarrassment of riches. A text export of one test of Mealographer yielded roughly 25 megabytes of data. There are a number of different ways eye tracking data can be interpreted, and the measures can be grouped into measures of search and measures of processing or concentration (Goldberg and Kotval, 1999):

Measures of search:

  • Scan path length and duration
  • Convex hull area, for example the size of a circle enclosing the scan path
  • Spatial density of the scan path.
  • Transition matrix, or the number of movements between two areas of interest
  • Number of saccades, or sizable eye movements between fixations
  • Saccadic amplitude

Measures of processing:

  • Number of Fixations
  • Fixation duration
  • Fixation/saccade ratio

In general, longer, less direct scan paths indicate poor representation (such as bad label text) and confusing layout, and a higher number of fixations and longer fixation duration may indicate that users are having a hard time extracting the information they need (Renshaw, Finlay, Tyfa, and Ward, 2004). Usability studies employing eye tracking data may employ measures that are context-independent such as fixations, fixation durations, total dwell times, and saccadic amplitudes as well as screen position-dependent measures such as dwell time within areas of interest (Goldberg, Stimson, Lewenstein, Scott, and Wichansky, 2002).

Because of the time frame of this investigation, the nature of the study tasks, and the researcher’s inexperience with eye tracking hardware and software, eye tracking data was compiled into “heat maps” based on the number and distribution of fixations. These heat maps are interpreted as a qualitative measure.

Continue reading →

A user-centered redesign of the Kent State SLIS site

Academic Papers, Projects, , , , ,

Note: This was originally created for an information architecture class – the project was to redesign the Kent State School of Library Science web site. You can also see a usability study of the site.

Executive Summary

The current Kent State University School of Library Science (SLIS) does not meet the needs of the department. This project outlines a plan and strategy for designing a new site. The new site will better communicate the department’s image and core attributes to the outside world and better meet the needs of users. This report covers the entire process, from research and project goals, through the development of a new design and how to measure success. Major recommendations include the use of a simple content management system (CMS), a new navigation structure and graphic design, and a few new content elements such as news, video, and podcasts.

Introduction

This report will cover the overall strategy for the redesign of the Kent State University SLIS web site, including the site’s audiences, the vision for the site, and analysis of the content and maintenance. Finally, recommendation are made for the content, information architecture, and design of the new site. The ultimate goal of this project is to create a coherent analysis and plan for the SLIS department to execute. The result will be a site that better projects the image of the department, better serves the users, and, if possible, makes the staff’s job a little bit easier.

Site content has been updated, but the organization and design of the site has been the same since 2000. The web has changed a great deal in the last 5 years, and the Kent SLIS site look and feel is not exactly cutting edge. The faculty and staff have voiced a desire to update the site, and there is anecdotal evidence that at least some students find the site lacking. Any new design must better address the needs of the site’s audiences and should better project the image of the department to the outside world. Also, the process used to update the current site is slow and unwieldy. The new site will solve three main problems: poor ease of use, an image that does not fit the department, and difficulty updating the site and communicating with users.

The process followed in creating this report has included requirements-gathering meetings with SLIS faculty and staff, content analysis of the current site, analysis of server logs, brainstorming sessions with Information Architecture Knowledge Management (IAKM) students, analysis of similar sites, academic usability research, the creation of persona, card sorting exercises, wireframing, prototyping and other techniques. The report will recommend additional steps such as formal usability testing be taken as well.

Continue reading →

Notes: Web site usability, design, and performance metrics

Academic Papers, Usability, , , , , , , ,

Palmer, J.W. (2002). Web site usability, design, and performance metrics. Information Systems Research, 13(2), 151-167.

In this study Palmer looks at three different ways to measure web site design, usability and performance. Rather than testing specific sites or trying out specific design elements, this paper looks at the validity of the measurements themselves. Any metrics must exhibit at least construct validity and reliability—meaning that the metrics must measure what they say they measure, and they must continue to do so in other studies. Constructs measured included download delay, navigability, site content, interactivity, and responsiveness (to user questions). The key measures of the user’s success with the web site included frequency of use, user satisfaction, and intent to return. Three different methods were used: a jury; third-party rankings (via Alexa), and a software agent (WebL). The paper examine the results of three studies, one in 1997, on in 1999, and one in 2000, involving corporate web sites. The measures were found to be reliable, meaning jurors could answer a question the same way each time, and valid, in that different jurors and methods agreed on the answers to questions. In addition, the measures were found to be significant predictors of success.

This is an interesting article because in my experience, usability studies are often all over the place, with everything from cognitive psychology and physical ergonomics to studies of server logs to formal usability testing to “top ten usability tips” lists. Some of this can be attributed to the fact that it is a young field, and some of it is due to the different motive fueling research (commercial versus academic). One thing in the article I worry about, however, is any measure of “interactivity” as a whole. Interactivity is not a simple concept to control, and adding more interactivity is not always a good idea. Imagine a user trying to find the menu on a restaurant’s web site—do they want to be personally guided through it via an interactive Flash cartoon of the chef, or do they want to just see the menu? Palmer links interactivity to the theory of media richness, which has a whole body of research behind it that I am no expert on. But I would word my jury questionnaires to reflect a rating of appropriate interactivity.

The most important impact of this study is that it helps put usability studies on a more academically sound footing. It is very important to have evidence that you are measuring what you think you are measuring. It would be interesting to see if other studies have adopted these particular metrics because of the strong statistical evidence in this study.

The most straight-forward metric, download delay, is also one that has been discounted lately. The thought is that with so many users switching to broadband access, download speed is no longer the issue it used to be. This is especially false for sites with information seeking interfaces, which are often very dynamic and rely on database access. No amount of bandwidth will help if your site’s database server is overloaded.