Posts Tagged ‘Usability’

Urban Usability - How walkable is your city?

Saturday, July 19th, 2008

Cleveland skyline from the Superior Viaduct I have a little project called Localographer, which you can use to create heat maps and find a house or apartment near your workplace, friends and relatives, or other place you’d like to be.  When I showed it to my brother he tried mapping out places in Boston and ran into a limitation - the interface doesn’t show you various transit options and it doesn’t make it easy to figure out the real cost and benefits of living in different places.

If you move to the suburbs, you might be able to commute by car but living by a train stop can be cheaper and easier.  In some neighborhoods you can get 10 different kinds of food in a 10 minute walk, in others you need to get in your car and drive a quarter mile to get anything to eat at all.

Adding features like this to Localographer means solving two problems - data and user interface.  I don’t have access to restaurant locations, transit stops, etc. and that sort of data can be expensive to get from commercial sources.  I could go the wiki route but that would require building an interface for users to contribute data and finding ways to make the data more reliable.

So in the mean time, if you want to get an idea of how walkable a potential neighborhood might be, take a look at Walk Score.  It’s a very cool site which has some of the features I’ve been meaning to add to Localographer - you can get a score for how livable the area around any address might be.

For example, my current neighborhood in California has a score of 74 out of 100.   Our house in Shaker Heights scores 62 out of 100.  Because any excuse is a good excuse to use a spreadsheet and a graph, I’ve plotted out the walkability of all the places I’ve lived using a Google Docs spreadsheet and the Interactive Time Series Gadget.  I wrote earlier about how you can embed any Google Doc or Spreadsheet into a blog post but Gadgets are even easier - just click the “Publish” button on the gadget and paste the Javascript code in the raw HTML view of your blogging software.

There are some issues with Walk Score, of course - for example Naples, Florida scores very high, but when I lived there I really missed having access to a car.  Most of the restaurants and shops along 5th Street and Tamiami Trail were out of my internship-funded price range.  I used to bike some distance to get to The Clock, a cheap diner.

All of this discussion is pointing toward a much larger question that I have been thinking about for a long time - I know how to study the usability of web sites and other software, but I wonder if anyone does usability studies of urban planning?  I’ve seen traffic flow studies and I know building codes have some basis in ergonomics and accessibility, but does anyone do observational studies of how people interact with different urban environments to figure out what works and what doesn’t?  Is there a Fitt’s Law of where to locate grocery stores compared to condos?

Sphere: Related Content

Problem with iPhone after 2.0 firmware update

Friday, July 11th, 2008

My wife tried to upgrade her iPhone to the 2.0 firmware this morning. Everything went fine until the phone got to the stage where it has icons telling you to lug into iTunes and the message “slide for emergency.”

Plugging in to iTunes gives us this message: “We could not complete your iTunes Store request. The network connection was reset.”

I finally found a thread on Apple’s discussion board with a workaround - keep the phone plugged in, and in iTunes, keep switching between the iPhone and another function like Music. You’ll keep getting the same error over and over, but supposedly it will work after a few minutes of trying.

After about 20 minutes of this ridiculousness (you can use the ‘m’ key and ‘i’ key to switch if you get tired of clicking), it finally looked like it was going through…

Only to tell me “iPhone activations are unavailable at this time.”

Clearly Apple underestimated demand, and their serves can’t handle the number of incoming requests. That isn’t much solace for my wife, who can’t use her phone today.

In the usability world, the technical term for this is “epic fail.”  We switched from Palm devices to the iPhone to avoid this sort of craziness.

I have to go to work. Guess I’ll try again later… anyone have any other workarounds?

Sphere: Related Content

Web Video Usability Review: South Park Studios

Monday, March 31st, 2008

After a few years of Youtube showing the world how to do video on the web, lots of traditional broadcasters and studios have started putting their content online. Part of the reason is to try to steal YouTube’s thunder - a more market-friendly tactic than just lawsuits. Many of these sites are trying to figure out an advertising model and make money, while others are obviously trying to get viewers more engaged by joining social networks, making mash-ups, etc.

But enough about their goals, what about user goals and experience? In web video the content may be king but usability is almost as important. If your user interface is difficult, confusing, or unpleasant, users will leave your site to get the content elsewhere.

So I’m going to try to do a usability review of various web video sites over the next few weeks. These won’t be formal reviews with user tests and cool eye-tracking heatmaps. Instead I’ll point out some user goals and hold up each site to the same rubrick.

The first site: SouthParkStudios.com

southpark-screenshot

So, what do users want out of web video? I can think of a number of scenarios: finding a particular clip or episode, watching recent episodes, sending a link to a friend or embedding a clip in a blog, and , well, just enjoying the show.

Selection

Score: 4 out of 4 points. This site has everything - every show from every season.

Finding Particular Videos

Method: I’m taking a cue from the creators of Friends - people don’t remember episode names. So I’ll be doing a Google search for the show name and “the one where” and taking the first relevant result. In this case it’s “the one where Ben Affleck has a relationship with Cartman’s Jennifer Lopez hand” (without quotes).

Score: 2 out of 4 points. The search fails, but a simpler query for “Ben Affleck” leads us right to the clips. The full episode is available.

Watching Videos

How easy is it to watch videos? What’s the quality?

Score: 4 out of 4. It’s immediately apparent what to click on to see an episode or clip. You can watch videos full screen and South Park’s animation lends itself well to compressed video. The navigation between episodes is pretty nice, with thumbnails of all episodes for that season along the bottom of the window.

Linking to Videos

Score: 3 out of 4 points. The URL for each clip and episode is available by clicking the “Share” button. Clips open up in the main window so if you can get the link like any other web page. The only lost point is the fact the episodes open in new windows - what is the point? It takes away my browser toolbar and any social bookmaking toolbars or extensions I might normally use.

Embedding videos

Let’s give it a try:

Score: 3 out of 4 points. Once again use the Share button to get the embed code. They lose a point for not allowing embedding of full episodes - they probably have good reasons for not wanting users to do so, but we’re only concerned about the user’s side of things right now.


Advertising

Score: 3 out of 4 points. Ads are shown before the video (for clips) or at two break points about halfway through (for full episodes). Commercials are short and don’t obscure video or interrupt the show more than normal TV commericals would. They lose a point, though, because of the lack of variety - I watched a few episodes and plenty of clips and only saw two different commercials, over and over again.

Audio Experience

I’m going in go with a slightly different scale this time: introducing the patented Bleeding Ear Scale of Web Video Volume.

You may have noticed that some TV stations play their commercials a little louder than the show. The theory I’ve always heard is that they want you catch your attention even if you get up to go to the fridge.

Score:

bleeding earbleeding earbleeding earbleeding ear

Unfortunately, most people don’t watch web video the same way they watch TV - they’re usually sitting much, much closer to the speakers or wearing headphones. The bone-shattering difference in volume between the video and the commercials on SouthParkStudios.com earned the site four bleeding ears.

Total score: 19 out of 24 points, with a special note to dive for the volume button whenever an ad is coming up.

Sphere: Related Content

The usability and design of two warning labels

Monday, January 14th, 2008

Usability and design aren’t just concerns for web developers.  They can make a real impact in the use and usefulness of physical products as well.

Warning labels are a great example - you can’t buy anything these days without some kind of warning label, and they are visual design elements intended to convey important information to buyers and users.  I ran into two great examples in the course of packing up and clearing out our house.

Example number one is from a big plastic storage tub.  It’s a great example of both usability and design, though the actual message might seem a bit silly.  Do people really need to be warned not to seal their children inside airtight containers?

A clear warning - do not store baby in plastic tub

It’s great from a design standpoint because it is clean, puts clear emphasis on the important diagram, and uses bright, attention-grabbing colors.  Any parents poised to place their toddler in the bin will not doubt see the label before recklessly replacing the lid.  It’s a good usability case because it conveys information very clearly and effectively-the silhouette kid is immediately recognizable and it uses common conventions such as the red circle and slash to mean “NO!”

The second example is…  well, strange and off-putting.  We might laugh at the thought of tupperwared toddlers but fireworks obviously pose some danger.  This series of warning messages from the back of a box of fireworks is, well…  take a look for yourself.  I recommend clicking on the image to zoom in in Flickr.

Strange fireworks warning label with creepy inhuman cartoon characters

So what could have been improved from a design standpoint?  For one thing, it would help if the coach and the two children in the second panel weren’t wearing what appears to be ghoulish, grimacing deathmasks.   They look like a cross between some misguided ventriloquist’s dummy and the clown that haunts the nightmares of every five-year-old child.  Cartoons caricatures can be very effective in warnings because we can remove unneeded visual detail to focus on what’s important and because people are accustomed to following short narratives in the style of comic strips.  But not if they are so ugly.

What are the usability problems?  Let’s start at the top.  The phrase “Common sense coach reminds you to…” isn’t quite as clear and gripping as “Warning: suffocation risk” in the first example.  The idea of using cartoons to illustrate each point is good, but the actual illustrations miss the point.  Without the text, do you think you could figure out the meaning of each of these?

For more examples of how usability and information design impact the real world, see The Design of Everyday Things by Don Norman.

Sphere: Related Content

Data Visualization with Maps

Monday, June 11th, 2007

One of the best ways to show relationships in data is also one of the oldest: maps. There are lots of cool, fun visualizations out there like topic maps and tag clouds, but sometimes they emphasize form over function (and usability). Maps can be a great choice, even if your data is not directly geographical.

Here’s one example: a map of the United States showing where people use the terms “soda,” “pop,” or “coke.”

You might think this one was a pretty obvious choice, but you could definitely imagine someone using a pie chart to show the total percentages instead, throwing out a ton of information in the process.

Here’s one that’s a little more clever: a map of the United States, which each state labeled by a country with the same GDP. from strange maps.

states-gdp.png

Now, you could argue with the precision of presentation since most people don’t know the exact GDP of Algeria off the top of their heads. But show them a table of figures and ten minutes later they still won’t know. This is a much more interesting and memorable presentation of the data.

Sphere: Related Content

Web Analytics and Usability

Friday, April 20th, 2007

I had the chance to catch a presentation by Matt Bailey about web analytics and usability. He made a great point - a lot of the kinds of problems that we look for with usability testing should show up in your web log data too, if you know how to analyze it.

Sphere: Related Content

Formal usability testing with eye tracking - Mealographer

Monday, May 15th, 2006

Usability Testing

Usability tests can be seen to fall into two general categories, based on their aim: tests which aim to find usability problems with a specific site, and tests which aim to prove or disprove a hypothesis. This test would fall into the former category. A search of the literature will reveal that tests looking to uncover specific usability problems often use a very small number of participants, coming from Nielsen’s (2000) conclusion that five users is enough to find 85 percent of all usability problems. Nielsen derived this formula from earlier work (Nielsen and Landauer, 1993). Although there is much disagreement (Spool and Schroeder, 2001), this rule of thumb has the advantage of fitting the time and money budget of many projects.

Use of Eye-Tracking Data

In terms of raw data, eye tracking produces an embarrassment of riches. A text export of one test of Mealographer yielded roughly 25 megabytes of data. There are a number of different ways eye tracking data can be interpreted, and the measures can be grouped into measures of search and measures of processing or concentration (Goldberg and Kotval, 1999):

Measures of search:

  • Scan path length and duration
  • Convex hull area, for example the size of a circle enclosing the scan path
  • Spatial density of the scan path.
  • Transition matrix, or the number of movements between two areas of interest
  • Number of saccades, or sizable eye movements between fixations
  • Saccadic amplitude

Measures of processing:

  • Number of Fixations
  • Fixation duration
  • Fixation/saccade ratio

In general, longer, less direct scan paths indicate poor representation (such as bad label text) and confusing layout, and a higher number of fixations and longer fixation duration may indicate that users are having a hard time extracting the information they need (Renshaw, Finlay, Tyfa, and Ward, 2004). Usability studies employing eye tracking data may employ measures that are context-independent such as fixations, fixation durations, total dwell times, and saccadic amplitudes as well as screen position-dependent measures such as dwell time within areas of interest (Goldberg, Stimson, Lewenstein, Scott, and Wichansky, 2002).

Because of the time frame of this investigation, the nature of the study tasks, and the researcher’s inexperience with eye tracking hardware and software, eye tracking data was compiled into “heat maps” based on the number and distribution of fixations. These heat maps are interpreted as a qualitative measure.

Methods

The goal of the study is to evaluate the usability of the web site and identify problem areas that might be improved.

Participants were solicited from a population of Kent State University graduate students in the School of Library Science (SLIS) and the Information Architecture Knowledge Management (IAKM) program via email. Five students volunteered, although one dropped out of the study at the last minute. Participants ranged in age from 25 to 35 with half female, half male. All had more than five years experience using computers and the web.

Participants were invited to the SLIS usability lab where they were informed of the procedure and asked for their consent. Once demographic data was collected, participants were calibrated on the eye tracking system and then asked to complete eight tasks using the web site. Participants were encouraged to “think out loud” while they attempted to complete each task, and their comments were recorded. Interaction with the web site was recorded by screen capture software and an unobtrusive eye-tracking device. Finally, participants were asked to fill out a short survey gaging their opinion of the site and it’s usability. Participants were free to stop at any time, and no incentive was offered for participation other than the opportunity to see the new lab and equipment.

Quantitative measures of usability include task completion, time required for task completion, and number of steps to completion. Participant comments and eye tracking data are used as qualitative measures.

Testing Difficulties

During the course of testing a number of problems arose with the eye tracking software, ClearView. At the conclusion of one of the four tests, the software locked up, making the computer unresponsive. Eventually it crashed, losing all test data for that participant. Consultation with a faculty familiar with the software confirmed that this is an unpredictable, but common bug in ClearView with no known workaround.

In addition, a deficiency was discovered in ClearView’s scanpath and hot spot visualization tools. Both overlay fixation data on top of screen shots of the web page the user was viewing. Unfortunately, the screen shots are not taken as the participant proceeds through each task. Instead the software remembers the URL of each page and then retrieves the screen shot at a later time. Many pages in Mealographer (and virtually any non-trivial web application) look different depending on whether or not a user is logged in, what the user has submitted in a form, and the presence or absence of cookies for session information. Therefore, many of the hot spot maps are laid over a version of the page dissimilar to what the participant had actually seen.

Findings

Task Completion Analysis

Overall, participants were able to complete the tasks 70% of the time. For the purpose of this evaluation, a task is considered complete only when the participant has found and used the expected feature for each task – often times, users were satisfied with their outcome, even though they had not used the most appropriate features. For example, two participants approached Task 1A by searching for a food they already knew was high in protein and low in fat, instead of finding and using the Healthy Food Search form. Table 1 displays the task list and completion rates. The tasks with the lowest completion rate were 1B, 6B, 7 and 8. It is interesting to note that the best-performing tasks include food search, meal entry, and user account functionality, which are the top items seen on the homepage when not logged it. This may be an indication of poor discoverability.

In general, the task completion rate indicates plenty of room for improvement.

Task Completion Rate 1. A. Think of a food that you like. Use this website to find out how much protein is in that food. 4/4
1. B. Try to find a food that is high and protein but low in fat. 1/4
2. Please find your way back to the Mealographer homepage without using the browser’s Back button. 4/4
3. Use this website to figure out how much fat and fiber you had at lunch today. 3/4
4. Sign up for an account. 4/4
5. Now that you have an account, please enter in what you had for breakfast today. How many calories did you have at breakfast? 4/4
6. A. Did you have more Calories at breakfast or lunch today? Find a way to figure this out. 3/4
6. B. Let’s say that you wanted to see a summary of your nutrition each day this week. Try to find a report that shows a week’s worth of information and look up your calcium intake. 2/4
7. Let’s say you wanted to try to have more than 25 grams of fiber each day. What’s the best way to make sure that you are with this web site? 2/4
8. If you were going to use this web site often, what would you do to make entering meals easier? If you can think of something, go ahead and do it. 1/4
Total 70%

Table 1: Task Completion

 

Completed Tasks

Incomplete Tasks

Number (recorded) 19 11
Average Time 1:56 2:37
Average Path Length 3.79 6.27

Table 2: Task Completion vs Performance Measures

 

Note that for the three participants with recorded times per task and path lengths, completed tasks took less time and involved a shorter path (see Table 2). This is logical, but with only three participants the correlation is not significant.

Individual completion rates, average times per task, and average path lengths are given in Table 3 along with user ratings of the site from the post-test questionnaire. It is interesting to note that user performance on task does not seem to correlate with user ratings (although with so few participants, correlations would not be statistically significant). The user with the highest completion rate gave the site the lowest ease of use score and lowest score overall, while the highest rating was given by a participant in the middle of the field for each performance measure.

Overall, the user ratings of the site were positive, but not outstanding. Although the ratings don’t correlate to the measures, many of the specific usability problems noted later in this report were noticed by the participants themselves as they used the site. Presumably changes to correct these problems would result in higher ratings.

Participant Completion Rate Avg Time per Task Avg Path Length Organization Rating Ease of Use Rating Design Rating
1 4/10 1:33 2.9 4 4 3
2 6/10 3:35 8.0 3 3 4
3 10/10 - 3 3 3
4 8/10 1:36 3.2 4 5 4
Total 70% 2:15 3.7 3.5 3.75 3.5

Table 3: Participant Performance and Site Ratings

 

Table 4 shows the task time and path length for each recorded participant broken down by task. Except for tasks 2 and 8, target path lengths for each task were set to the smallest path that would achieve the goal plus one, and target task times were set somewhat arbitrarily to 2 minutes. Task 2 was much simpler than the other tasks and task 8 was open-ended, allowing participants to enter favorites, usuals, or both. It is difficult to set a task time goal for an interactive web site without data from previous tests. The amount of time a user might want to spend on a particular activity before becoming frustrated or giving up may very based on a number of factors including user motivation, user enjoyment, and web site stickiness. The target of two minutes was chosen because it was thought to be fairly aggressive, meeting the goal of the project to make diet tracking quick and easy.

 

Participant 1 Participant 2 Participant 4 Average Target
Task Time Path Time Path Time Path Time Path Time Path
1. A. 1:10 3 1:55 4 1:33 4 1:33 3.67 2:00 3
1. B. 0:38 1 1:29 4 1:11 2 1:06 2.33 2:00 3
2. 0:08 2 0:17 2 0:17 2 0:14 2 0:30 2
3. 1:56 3 4:35 3 2:17 4 2:56 3.33 2:00 4
4. 1:18 4 1:19 4 1:56 5 1:31 4.33 2:00 4
5. 1:25 3 4:56 3 2:53 3 3:05 3 2:00 4
6. A. 1:51 3 5:26 14 1:52 4 3:03 7 2:00 3
6. B. 1:23 3 5:59 15 0:36 1 2:39 6.33 2:00 2
7. 1:16 4 3:29 8 1:30 3 2:05 5 2:00 5
8. 2:48 3 6:20 23 1:58 4 3:42 10 - -
Total 13:53 29 35:45 80 16:03 32

Table 4: Performance Measures by Participant

 

Items in Table 4 in bold denote tasks in which a participant had not met the target, and those with gray backgrounds represent successfully completed tasks. Looking at average performance, only tasks 1B and 2 completely met all objectives. It would be reasonable to look for specific usability problems in the tasks for which each participant missed one or more target, and many are addressed in the Specific Usability Problems section below.

Hot Spot Analysis

In Illustrations 5, 6, and 7, hot spot maps of the Mealographer home page are shown for three participants (one, two and four, respectively). These images show the relative number of fixations by color, with green meaning at least one fixation and red meaning three or more, with fixations length set to a maximum of 100ms.

 

Hot spot map for Mealographer user 1

Illustration 5: Hot spot map for participant 1

Hot spot map for Mealographer user 2

Illustration 6: Hot spot map for participant 2

Hot spot map for Mealographer user 4

Illustration 7: Hot spot map for participant 4

These hot spot maps illustrate some of the observations that can be made with eye tracking that might be missed by other usability testing methods. For example, participant one did not read the light blue help boxes at the bottom of the page whereas two and four did. The participants fixated on many of the red “* required” labels, but only briefly - this may mean that they served their purpose in alerting users without causing confusion or additional concentration. Note the large red areas under the drop-down boxes for “Month” and “Question.” Users had to fixate more on these controls than on many of the text fields.

Scanpath of user 1 looking for Coffee

Illustration 8: Scanpath for participant searching for “coffee”

Scanpath of an expert user looking for Tea

Illustration 9: Scanpath for expert searching for “tea”

Scanpath of user 4 looking for Coffee

Illustration 10: Scanpath for participant searching for “coffee”

Scanpath Analysis

The scanpaths in Ilustrations 8, 9 and 10 show the result of users entering in a search term on the meal entry form and searching for the proper item to add to their meal. The two on the left represent searches for “coffee” by two participants, novice users of the site. The one on the right is a search for “tea” by the researcher, and can be considered an example of expert use. Note that the paths of both novice users are both much longer than the expert, and that the convex hull area would be larger as well, despite the fact that the desired item is located in a similar position on all three. The novices need to gaze at each item on the list in order, and require more gazes on likely items in order to make a decision. In all three cases, the long, spread out gaze paths indicate that desired items are not near the top of the results, and are not readily apparent. Improvements should be made to the search engine and the results display, perhaps highlighting search terms within each line.

Specific Usability Problems

A list of specific usability problems has been compiled from the participant performance results in Table 4, observations made during the test by the researcher, and participant comments. Problems are organized by task.

Task 1A – Food search

Search results do not always match user expectations. Multiple participants expresses confusion or had to refine their search to get to the food they were looking for. As the tagging system grows, it will help to address this issue.

Search results are hard to scan. Participants seemed lost in all the text in the results. This can be addressed by highlighting the search terms within the results and visually dividing items from each other.

The ordering of nutrients on food pages might not be intuitive. The food pages were designed to match the ordering used on product packages, but some users needed extra time to find what they were looking for. One participant said she expected them to be in alphabetical order.

Task 1B – Healthy food search

Users did not notice the healthy food search function at first. Three users tried using the food search to search for a food they already knew was high in protein and low in fat, at least at first, and another was satisfied with the food he had just searched for by name. The healthy food search could be given more prominent placement, perhaps above the scroll on the search results page.

Task 3 – Meal entry using the quick form on the homepage

One user did not find the meal form on the homepage. Participant 1 used the food search instead, explaining that she had only had one thing to eat. Since the other found the form easily, action might not be needed to correct this.

Results for each food entered in the form do not match user expectations. One participant, for example, entered “rice” but did not find plain rice in the dropdown on the next page. This is another search engine difficulty.

Users could have refined their terms, but did not see a way to do so. In fact they would have had to go back to the homepage and submit the new terms. An entry box could be added to the results page to allow users to refine their terms.

Some users left some fields blank, and did not get useful results. One user left all the “Units” fields blank and the results showed a meal with no nutrition. It took some time for that user to figure out the cause of the problem. Validation hints could be added to the form, and units could default to one instead of blank or zero.

Task 4 – Account creation

Meals entered by visitors could be lost if they do not go directly to create a new account. One user missed the link to create an account from the meal page and went to the homepage first. This is a perfectly valid action. The ID of the temporary meal record in the database should be associated with the user session so visitors do not have to re-enter meals later.

Task 5 – Meal entry

It is not immediately clear how to use the meal entry form. Three users tried to click on the notepad at first, expecting it to work similarly to the quick meal form. Although both did figure out to use the search form to find and then add items, this could be improved. Currently when the form first loads, the search results frame is blank. An arrow graphic and some explanatory text might make use of this form more clear. It might also make sense to have only one meal entry form, mixing the strengths of each.

Users did not always try to modify search terms to improve results. At least one user suggested good modifiers when thinking out loud, but did not try any of them. There is text in the results that suggest trying again, but it could be made more clear.

Food search results were difficult to scan. Participants seemed to take a long time finding the item they wanted on the list (see Illustrations 8, 9, and 10 for scanpath diagrams). One participant suggested adding bullet points to better mark items in the list. Search terms could also be highlighted in the results.

Users are not sure how to enter complex or compound food items. One user tried “salad” and “caesar salad” but compound items with many variable ingredients like salads and sandwiches are left out of the database. Some instructions could be added. This might also be solved by the addition of a food or recipe entry system.

Some users left some fields blank, and did not get useful results. Like the quick meal form, validation hints could be added to the form.

Task 6A – Use of daily report

Some users didn’t think to look for a report system. Three users used pen and paper to compare meal totals instead, although two did later find the daily reports. The existence and functionality of the reports could perhaps be made more clear, and additional links might make the reports more discoverable.

Reports (and many other sections of the site) are not identifiable from browser history.

Participant 2 had a particularly difficult time finding the reports using the browser’s history function. Page titles were very similar, and he tried nine different links before trying another tactic. This is important because it means many pages would not make good bookmarks either. Better, more succinct page titles need to be used on interactive pages.

Reports cannot be accessed when not logged in. In the course of using the browser history, participant 2 accidentally logged himself out without noticing. This made it almost impossible to find the reports, and when he did find the reports they did not have any data. Care was taken to ensure users could not alter other users data, or make entries without logging in, but dynamic pages such as the reports should clearly remind users to log back in rather than displaying zeros. Once that is done links to the reports could be added to the tour page.

Task 6B – Use of weekly report

Users were not immediately aware of the weekly report. The control to switch to the weekly report is located below the scroll on the report page. It could be moved up or a simpler control could be added to the top of the page.

Task 7 – Goal creation

Users did not think to look for goal-setting functionality. Two users reported being happy just using the reports to track progress, though one later did find the goal functionality. Another user overlooked the link to create a goal at least once before finding it. Goals should be better integrated into the report pages.

The list of goals was confusing when no goals were listed. A new user looking at the list before entering sees the column headings but no indication that this is list with zero items. The list should not appear when no goals exist.

Some terms are unfamiliar to users. One participant did not know what “DRI” and “Daily Value” meant when entering a goal. Tooltips or links to a FAQ could be used to help inform users.

Task 8 – Use of usuals and/or favorites

The list of favorites was confusing when no favorites were listed. This is also true for usuals. The list should not appear when no favorites exist.

Favorites and usuals were not discovered by most participants. Three participants either did not think to look for such options or did notice them while using the site. Favorites and usuals need to be better integrated into the rest of the site. It might be helpful to indicate existence of favorites on meal page, even if user has no favorites. Also, an “add this food to my favorites” link could be added to food pages.

 

 

 

References Cited

 

Gibson, J. J. (1979). The Ecological Approach to Visual Perception. Hillsdale, NJ:Lawrence Erlbaum Associates.

 

Goldberg, J.H., Kotval, X.P., (1999). Computer interface evaluation using eye movements: methods and constructs. International Journal Of Industrial Ergonomics 24, 631–645.

 

Goldberg, J.H., Stimson, M.J., Lewenstein, M., Scott, N., Wichansky, A.M. (2002). Eye Tracking in Web Search Tasks: Design Implications. Proceedings of the 2002 symposium on Eye tracking research & applications. New York: ACM Press. pp. 51-58.

 

Golder, Scott A., Huberman, Bernardo A. (2006). Usage patterns of collaborative tagging systems.

Journal of Information Science, Vol: 32, Issue: 2, April 2006, pp. 198-208

 

Lindgaard, Gitte, Fernandes, Gary, Dudek, Cathy, and Browñ, J. (2006).

Attention web designers: You have 50 milliseconds to make a good first impression. Behaviour and Information Technology, Volume 25, Number 2, Number 2/March-April 2006, pp. 115-126(12).

 

Nielsen, Jakob. (2000). Why You Only Need to Test With 5 Users. Alertbox March 19, 2000. Retrieved May 15, 2006, from http://www.useit.com/alertbox/20000319.html

 

Nielsen, Jakob and Landauer, Thomas K. (1993). A mathematical model of the finding of usability problems. Proceedings of the SIGCHI conference on Human factors in computing systems. April 24-29, 1993, Amsterdam, The Netherlands. pp. 206-213.

 

Norman, Don. (2004). Affordances and Design. Don Norman’s jnd.org. Retrieved May 15, 2006, from http://www.jnd.org/dn.mss/affordances_and.html

 

Renshaw, J.A., Finlay, J.E., Tyfa, D., Ward, R.D. (2004). Understanding visual influence in graph design through temporal and spatial eye movement characteristics. Interacting with Computers. Vol: 16, Issue: 3, June, 2004. pp. 557-578.

 

Spool, J. and Schroeder, W. (2001). Testing Websites : Five Users is Nowhere Near Enough. CHI ‘01 extended abstracts on Human factors in computing systems. New York: ACM Press. pp. 285-286.

 

Stafford, Tom and Webb, Matt. (2005). Mind Hacks: Tips and Tools for Using Your Brain. Sebastopol, CA:O’Reilly Media, Inc.

 

 

 

Sphere: Related Content

A user-centered redesign of the Kent State SLIS site

Thursday, December 15th, 2005

Note: This was originally created for an information architecture class - the project was to redesign the Kent State School of Library Science web site. You can also see a usability study of the site.

Executive Summary

The current Kent State University School of Library Science (SLIS) does not meet the needs of the department. This project outlines a plan and strategy for designing a new site. The new site will better communicate the department’s image and core attributes to the outside world and better meet the needs of users. This report covers the entire process, from research and project goals, through the development of a new design and how to measure success. Major recommendations include the use of a simple content management system (CMS), a new navigation structure and graphic design, and a few new content elements such as news, video, and podcasts.

Introduction

This report will cover the overall strategy for the redesign of the Kent State University SLIS web site, including the site’s audiences, the vision for the site, and analysis of the content and maintenance. Finally, recommendation are made for the content, information architecture, and design of the new site. The ultimate goal of this project is to create a coherent analysis and plan for the SLIS department to execute. The result will be a site that better projects the image of the department, better serves the users, and, if possible, makes the staff’s job a little bit easier.

Site content has been updated, but the organization and design of the site has been the same since 2000. The web has changed a great deal in the last 5 years, and the Kent SLIS site look and feel is not exactly cutting edge. The faculty and staff have voiced a desire to update the site, and there is anecdotal evidence that at least some students find the site lacking. Any new design must better address the needs of the site’s audiences and should better project the image of the department to the outside world. Also, the process used to update the current site is slow and unwieldy. The new site will solve three main problems: poor ease of use, an image that does not fit the department, and difficulty updating the site and communicating with users.

The process followed in creating this report has included requirements-gathering meetings with SLIS faculty and staff, content analysis of the current site, analysis of server logs, brainstorming sessions with Information Architecture Knowledge Management (IAKM) students, analysis of similar sites, academic usability research, the creation of persona, card sorting exercises, wireframing, prototyping and other techniques. The report will recommend additional steps such as formal usability testing be taken as well.

Audience and Vision

Methods

Information about the site’s major audiences and the vision for the site were gathered through three methods. First the current site was carefully reviewed for content and organization. Second, a requirements meeting was conducted 9/14/04 with members of the department staff, including Rick Rubin, SLIS Director, Cheryl Tennant, Student Services, Anna Gower, Senior Secretary, and Rhonda Filipan, Academic Program Coordinator. In this meeting we discussed both who the site should serve and what the department want the site to convey. Third, a number of brainstorming sessions were conducted with IAKM students in the Foundational Principles of Information Architecture class with professor David Robins. This section presents the findings of these efforts and lays the groundwork for the site as a whole.

Audiences and user groups

This site must address the needs of a number of different user groups, including:

Current Students – This web site is, after all, the first place that many students will look for information about department policies, classes, faculty and staff, and other information. It can serve as an important means of communication between the department and students along with mailings and email.

Possible subgroups include full-time/part-time, new students/returning students, and students attending the various programs or attending at the different campuses. We believe that these distinctions should be addressed, but that often all students will be performing similar tasks and searching for the same kinds of information. New students will be given special attention, since they may have specific information needs. The different campuses (Kent vs. Columbus) will not be as separated as they are in the current site, although it is important that users not need to guess which information is campus-specific.

Prospective Students – Although the department did not identify recruiting as a problem area, the web site is a natural vehicle for prospective students to find information about the program, how to apply, deadlines, and other information. If the site addresses prospective students we can achieve two goals: first, bringing in a larger, more diverse pool of applicants, and second, saving time spent answering common questions and providing a central location to point prospective students to for all their needs.

Possible subgroups include recent graduates/returning professionals, and applicants could be grouped by the program they wish to pursue (for example, the K-12 program vs. the MLIS/MBA degree). We believe these groups will have a number of tasks in common and can be best addressed as lower-level subdivisions of the prospective student area.

Alumni – The department maintains active contact with alumni and there are alumni events, dinners, a newsletter, and an Alumni and Friends Council. None of this is immediately apparent from the current web site, although there is some alumni information under the “People” section. Although not a primary audience like current and prospective students, the Alumni are an important group with very different tasks and information needs that should be addressed.

Faculty and Staff – There is some information on the site that would be of particular use to faculty and staff, for example the “Request Software” form. The most important task that some faculty and staff members have, however relate to adding and updating content on the site. Because we recommend using a content management system (CMS) for the site, it is important to directly the needs of those who will be updating the site and handle issues like access control.

Other LIS Programs and Professionals – This is in some ways a catch-all category for other users with more general, informational tasks. Individuals in other LIS programs may be interested in the department’s mission statement, contact information for a specific faculty member, etc. Researchers, educators and journalists may also use the site to find description and contact information.

Persona

One way to specifically address the needs of users is by developing and applying persona—architypical users that help put a face on who will be visiting the new site. For this report, we give the example of Sally, a member of the prospective student audience.

Persona: Sally 25 years old / female / single / Stow, OH
“I want to do something in science or academia, but I’m sick of short-term jobs and moving around the country.”
Current profession: manager at Wal-Mart
Background: Sally graduated with a BA in Biology in 2002 and has since had a number of short-term positions in and out of her field. She’s worked as a lab tech, as a park guide, and even worked for a large zoo (mostly shoveling manure). In between “real” jobs, she has to do something to pay the bills-hence he current position at Wal-Mart.
Goals: Sally has heard about science librarianship and wants to find out more. She’d like to stay in Ohio but is willing to move. She wants to find a good school that will give her good job prospects.
Tasks:
1) Find out a little more about library science to see if it looks like something she wants to do.
2) Find out about requirements - GPA, GRE, etc.
3) Find out about deadlines for application.
4) Make sure a LIS degree from Kent can get her a job-what’s the placement rate?
5) Look at courses to see if any sound interesting.
6) Find a professor who teaches a science library course and get contact information.
7) Apply for admission.

Vision

Now that we have established who will be using the site, it’s important to determine what the department wishes the site to convey. A site can only convey so much information at once, and it is important to pare down the attributes we are trying to communicate so that they are clear and effective.

Most of this section comes from the requirements meeting, where the staff in attendance agreed that the site should convey the following.

Cutting Edge – this is a library and information science program after all, the faculty are conducting research and teaching methods on the cutting edge of the field, and this site should reflect that. Kent’s SLIS program that leans more toward technology than the humanities. This has implications for the site design—the site must be visually modern and striking, and employ an advanced information architecture and standards-based coding. The site should be accessible and printable.

Professional – Kent State’s SLIS program is specifically geared toward training professionals to go out into the world and work, whether it be in a public library, K-12 or even the private sector. Research and grants are not as important as quality teaching and placement. The site should project a very professional image. This should modify the Cutting edge aspect of the site somewhat—that means no fancy, pointless Flash animations—we are not here just to show off.

Caring – The library profession requires constant interpersonal interaction and few librarians get into the field because of money—instead they wish to educate others, spread knowledge, and support the democratic process. Kent State’s program is a nurturing place, a community, and the site should reflect that. An important aspect of this attribute that must be stated is a respect for Diversity. The panel members specifically mentioned their wish to reach out to groups and communities that are not currently well represented in the field.

These three attributes should be supported when making decisions on content and features, information architecture, and graphic design. How do we integrate all three of these attributes? Providing streaming video of professors would be a perfect example. Video is cutting edge, but can be done very professionally, and puts a face, voice and personality to the list of names on the faculty page.

Another example would be sample podcasts of lectures. This would have to be done on a volunteer basis, perhaps with permission of the university, but podcasts are one of the hottest new media on the web, allowing people to download audio programs to their MP3 players and listen at their leisure. Reputable organizations like NPR are making podcasts available and they also give a much more personal introduction to a course than a syllabus.

One more attribute should be mentioned, although this one is directed at specific user groups.

Authoritative – This is the official site of the program, and students should be able to trust the information they find here. That has important implications: content must remain up-to-date, content must be under editorial control, and content should be reasonably complete. It also means that highly interactive features like message boards, student blogs, and chat should probably be avoided if they can’t be kept under strict control.

Content Analysis

Content must be driven by two factors: what do audiences want to know, and what does the organization want to communicate? The SLIS site’s audiences are described in detail above. The new site’s content will be primarily composed of current content, reorganized, with a few new features. We believe that this fits well with the scope of the project.

The findings below are a result of careful analysis of the current content and navigation structure, other SLIS program web sites, and traffic logs.

Current content

Map of current siteThe current content supports many of the user tasks identified and does a fair job of meeting the organization’s goals, if the site design and organization does not. This makes sense because of the organic way in which the site has grown: most content has probably been added as new programs, facilities, faculty, etc. appeared or as staff members noticed users looking for it. Most of the changes we propose involve rearranging, reorganizing, and presenting the content more effectively.

The current content is divided into seven major categories. A sitemap diagram shows the first two levels of organization (Attachment 1). Other, similar SLIS programs have similar content. (See attachment 2). ….

New content

We recommend five major pieces of new content for the site:

1) News – The current site has a small news page, but it is not very visible, not regularly updated, and primarily consists of press releases. Creating a more visible and interesting news section can serve a number of site goals and a number of user groups. Regular notices about faculty publications and scholarships will look attractive to prospective students. Current student and other users that repeatedly visit the site, will notice the regular updates—credibility is enhanced by timeliness. Also, this will help serve the departments communications needs, working along side mailings, emails, and other methods of contact. It is important to note that this requires an additional commitment for the department.

Since this is not a portal, there is no need to update daily or on-the hour. The department should try to update the news section at least once a week, either with a press release-type item, important dates and deadlines on the academic calendar, new content available on the site, or even links to interesting library and information science articles in magazines or journals. If the CMS ultimately used for the site supports it, RSS feeds should be made available to highlight the department’s awareness of new and interesting technologies.

2) About library and information science – This content would be targeted specifically at prospective students. Although a large number of incoming students are already in the field, many come from different undergrad backgrounds or are looking for a later career change. The department has expressed a desire to attract a wider and more diverse field of applicants, and information about the field—what is studied, what careers in library and information science look like, descriptions of different programs—would help a great deal.

Unlike news content, this section would require a bit of work up front but could be useful without frequent updates. The new content would need to be written by someone in the department with an understanding of the different programs and the field. It’s possible the department already has flyers or mailings that could be used here. It is important to consider the main attributes listed above when creating this content—writing should be professional but personal. We are not selling library science, we are describing and explaining it.

3) Video – We highly recommend adding appropriate video to the site because it helps support the sites’ core attributes and goals. This idea was brought up during the requirements meeting. Right now each faculty member has the standard C.V. and list of interests. Adding a video clip where the faculty member introduces themselves and talks a little bit about their interests and classes would add a much more personal touch. In addition to pointing out the technological savvy of the department a simple, polished video presentation with decent lighting and sound would look very professional. Other areas that could benefit from a video introduction would be the distributed learning classroom, the Reinberger Children’s Library Center, and other facilities. It is important to constrain video to only subjects interesting to users, specifically prospective students.

Creating of this content would require further investigation, but it is very likely the department already has the resources to do this (or if not, can access resources elsewhere in the university). Faculty videos should be fairly simple to produce, but other videos might require a little more work to script and edit. Technical issues regarding video format and bandwidth should be left up to the web developers with the guideline that the end result be as easy to use as possible.

4) Existing documents not currently on the site – One measure of a site’s credibility is breadth in coverage. It is hard to convince current students that the web site is a “one-stop shop” for information if they find items missing. This includes additional forms, pamphlets, mailers, etc. produced by the department. At the very least, PDF copies of other publications should be made available on the web site. Anything produced for public release should be on the site, with exceptions made for documents that are confidential, must be placed elsewhere (on the university home page, Vista, etc.), or are otherwise inappropriate.

5) Podcasts – Although audio on the web is nothing new, podcasting has grown quickly since it’s start in 2003 and Apple’s addition of podcasts to iTunes in June. “Podcasting” is a misleading term, since they are not limited to iPods and do not employ traditional broadcasting. In essence audio is recorded and then published online where users are able to subscribe, download, and listen at their leisure. We believe podcasting a few lectures each semester would give a big boost to the department’s cutting-edge image and might create interest in the department and classes in new ways. Faculty members would be encouraged to take a look at their lesson plans and pick a class they wouldn’t mind having recorded and made available—the first day of class, for example, or a lecture on a subject they’ve done research on.

This content presents a few challenges that would have to be overcome. Any participants being recorded would need to know ahead of time about the recording and volunteer. The university’s legal department should be consulted. Recording should not be too difficult, with computers available in almost every classroom, although a microphone and software might need to be purchased (or borrowed). The web developers will need to make sure that uploading and publishing of podcasts is possible through the site’s CMS.

Publishing Content

The current method of publishing content on the site must be improved in order to support the goals of the department and reduce workload. Currently different sections of the site are the responsibility of different staff members. When an update needs to be made to an existing page, the staff member in charge prints out the page, makes the changes, then hands it to the network administrator. New content is sent to the network administrator as well. This results in a time lag to get any changes on the site and some duplication of efforts.

In order to keep the site up-to-date, support new content mentioned above, and cut down on the amount of work, we highly recommend the new site use a CMS instead of static web pages. The network administrator’s time is often limited (and upcoming organizational changes may leave the department without their own administrator). Training a staff member to update and manage a static HTML web site would take time and none of those present at the requirements meeting expressed real interest.

We believe one staff member should be designated as the main contact for web site updates with others serving as backups. The main contact would be in charge of the news section and most updating, although different staff members may still be in charge of their own section.

The CMS must meet the following requirements:

1) Stable – once the web developers have set it up, it must require minimum maintenance work. That means we should limit our search to well-regarded commercial software or mature open-source applications.

2) Affordable – some commercial CMS’s cost tens of thousands of dollars. This system must fit well within the department’s budget.

3) Simple – This site does not require many of the features and functionality that some CMS’s provide. For example we have no need for message boards, live chat, student logins, and blogs. It should be as easy as possible for the staff to update the site.

4) Functional – the CMS must support the site’s requirements, meaning we must be able to update a front-page news section, post files such as PDFs and Word documents, link to video and audio, and support the desired information architecture.

Site Redesign

Requirements

We have three overarching requirements. First, the department’s web site must support the tasks of user groups we have identified:

  • Current students
  • Prospective students
  • Alumni
  • Faculty and staff
  • Other programs and professionals

The five groups have one thing in common: their task sets are by and large limited to information-seeking. Some times this is part of a larger task, such as prospective students wishing to apply of admission or current students wishing to select classes. The SLIS website, however, is used in information gathering stages of these tasks rather than final execution.

Second, the site must actively project the key attributes of the department:

  • Cutting Edge
  • Professional
  • Caring

Finally, the site must support the department’s need to communicate effectively. This includes different kinds of communication to different user groups. For example the department may wish to use the website to both answer current student’s questions about graduation while also attracting prospective students doing general web searches.

Functional Requirements

We believe the requirements listed above imply that the new site:

  • Contain a user and task-driven primary navigation structure, with other parallel navigation structures a possibility.
  • Contain content to address those users’ needs, including the current content and new content identified above.
  • Be designed to express the department’s attributed visually.
  • Have a CMS so that updating and managing content is much easier.
  • Support serving and organizing PDF documents, audio, video, and other files.
  • Support modern, accessible, standards-based markup that is search-engine friendly.

Goals and Measurement

Without measurement we cannot be certain that the new site has met our goals. Three key areas of interest brought up during the requirements gathering meeting included:

1) Ease of use – Users must be able to find what they are looking for easily. This was phrased mostly in terms of task completion, although other measures (such as time to complete or steps to complete) may be useful as well. We recommend running a formal, but small usability test with the current site and then with the new design, comparing results. For the test, four or five members of each group should run through a small number of representative tasks. If it is necessary to cut corners due to time constraints we would recommend running the test with the current student user group, since they would be the easiest to recruit and work with. For each task we will record and compare the completion rate, the average time per user, and the average path length (number of clicks) to completion. It would be nice to run a usability test with content creators (staff members) as well, but we would have no earlier system to compare to and (depending on the CMS chosen) less flexibility to make changes in the future. One other measure might be the number of calls and emails sent to staff about information already available on the site. Staff members could be asked to keep a count of such contacts for a few months while the current site is in operation, then keep a count after the new site has come online to compare.

2) User opinion – Users should be polled about the old site and the new site to determine two things: do they like it, and what does it say to them? The former can be measured with a Likert-type scale and the latter by word association. Users could be asked to simply list words to describe the site, or pick words from a list, and the results matched against the site’s attributes. This could be folded in as part of the usability study.

3) Comparison to other SLIS sites – this is a broader area but could be measured in several ways. The first would be a survey asking appropriate user groups (prospective students for example) to look at and then rate the Kent State SLIS site versus several others. Another measure could be search engine ranking—target search phrases would be run on a popular search engine like Google and the site’s page rank recorded. This last method would be easier to do and could measure the new site versus the old, but would be a less direct measurement. Another measure might be an increase or increase in diversity of the applicant pool, although it is hard to eliminate external variables from that measurement.

Site Functions

Navigation

There are a number of ways that a web site can be organized, but the most important concern for an informational site such as this is that users are able to find what they are looking for. This means that a generic, org-chart navigation will probably not be very successful—most users will not know or care how the department is organized. Many department web sites a hierarchical navigation structure using common subject areas such as “programs,” “courses,” and “people.” Three schools identified by the department as similar or competing institutions, organize their sites into major sections this way:

University of Kentucky School of Library and Information Science: University of Pittsburgh Department of Library and Information Science Wayne State University Library and Information Science Program:
Main Navigation Links:
General Info Academics About Us
Academics Degrees Degrees & Certificates
Admissions People Admission & Fin. Aid
Courses About DLIS Courses & Schedules
People Services & Policies
Facilities People & Groups
News Calendar & Events
Jobs & Resources
Forms
Student Handbook
LISAA (alumni association)
Sidebar Links:
Mission Prospective Students
Newsletter Current Students
Our College Faculty & Staff
UK Libraries Alumni & Visitors
Student Info Update Research Projects
Contact Us The Fine Institute

From these examples two things are clear: first, it is common to include more than one method of navigation (with both a main navigation bar and a sidebar of links on the homepage), and second, that these departments tend to be organized into sections (by subject) similar to the current Kent SLIS site.

New navigation structure diagram

Since we have such well-defined user groups, however, we do not believe that is the best structure for the primary navigation scheme. Instead, we will use a faceted organizational scheme with the primary facet addressing each user group individually and a secondary navigation structure by subject.

Primary Navigation: User groups and tasks

Wireframe mockup of site designThe first four user groups are easiest to address directly, with Prospective Students and Current Students the primary and Alumni and Faculty and Staff as secondary target audiences. The more nebulous Other Programs and Professionals group is probably least important and is probably least likely to self-identify. The home page and user-group based navigation should reflect the relative importance of the different audiences, with the first two at the top of the list, taking up more screen real estate. The second level navigation will be by task. So, for example, the user might go to the home page, identify themselves as a current student, and from there choose a link labeled “find courses” in the second level. (See attachment)

Wireframe mockup of site designOn the homepage we will have a navigation area with a section for each of the user groups. Within each section will be a few top likely tasks with direct links. Clicking on the sections takes users to a landing page directed specifically at them, with the full task list, and each task-level link will either hit a further landing page or go direct to the resource that fulfills the task.

Secondary Navigation: By Subject

Wireframe mockup of site designThere will be a secondary navigation area or sidebar with the traditional subject-category scheme, similar to the current navigation structure. The subjects will include:

  • About SLIS (includes current About Us section)
  • Library Science (includes new content about the field)
  • Programs and Degrees (includes current Programs section)
  • Courses (includes current Courses section with any new content)
  • People (includes current People section)
  • Resources (includes current Facilities and Links sections)

Labeling

Wireframe mockup of site designIt is extremely important that labeling follows or accommodates user language as opposed to unfamiliar department or specialist language. Each of the user groups should be addressed directly. So it is less appropriate to use jargon with prospective students than current students. The page for the “12-12-12 Distance Degree Program” is a good example: a link for prospective students should identify it is “Distance Degree Program,” “Distance Learning,” “Off-campus Master’s Degree,” or something similar, since “12-12-12” will not mean anything to most incoming students. Links to the same page from the current student pages could use the “12-12-12” label, so long as it is what students use to refer to it.

We will also strive for clarity in labeling. The site’s graphic design must be able to accommodate longer labels and link titles where appropriate.

Design Ideas

Design mockup

Design mockup

Search

Search is extremely important in any site with sizable content. Although we take great pains to organize the site so that users can find what they are looking for quickly and easily, many will prefer to simply type in keywords rather than navigate.

Further, a large number of the prospective student group will come to the site by search engine rather than through the university, or department home page. Making it easier to find the department, or department pages, from the outside world can only help. That means that it is imperative that:

  1. The CMS chosen or developed for the site must create standards-complaint, readily-indexable web pages
  2. The CMS must create search-engine friendly URLs
  3. The site’s navigation must make it clear the context of the each page within the site, in case a user comes to that page directly (this also rules out the use of frames).

We do not recommend creating a search engine from the ground up for this site. That would be a large scale project in and of itself, and the return would probably not be worth the investment. Instead we recommend using either the CMS’s built-in search capabilities or a popular engine like Google, or perhaps both (on an “Advanced search” page).

Sphere: Related Content

Notes: Web site usability, design, and performance metrics

Sunday, July 3rd, 2005

Palmer, J.W. (2002). Web site usability, design, and performance metrics. Information Systems Research, 13(2), 151-167.

In this study Palmer looks at three different ways to measure web site design, usability and performance. Rather than testing specific sites or trying out specific design elements, this paper looks at the validity of the measurements themselves. Any metrics must exhibit at least construct validity and reliability—meaning that the metrics must measure what they say they measure, and they must continue to do so in other studies. Constructs measured included download delay, navigability, site content, interactivity, and responsiveness (to user questions). The key measures of the user’s success with the web site included frequency of use, user satisfaction, and intent to return. Three different methods were used: a jury; third-party rankings (via Alexa), and a software agent (WebL). The paper examine the results of three studies, one in 1997, on in 1999, and one in 2000, involving corporate web sites. The measures were found to be reliable, meaning jurors could answer a question the same way each time, and valid, in that different jurors and methods agreed on the answers to questions. In addition, the measures were found to be significant predictors of success.

This is an interesting article because in my experience, usability studies are often all over the place, with everything from cognitive psychology and physical ergonomics to studies of server logs to formal usability testing to “top ten usability tips” lists. Some of this can be attributed to the fact that it is a young field, and some of it is due to the different motive fueling research (commercial versus academic). One thing in the article I worry about, however, is any measure of “interactivity” as a whole. Interactivity is not a simple concept to control, and adding more interactivity is not always a good idea. Imagine a user trying to find the menu on a restaurant’s web site—do they want to be personally guided through it via an interactive Flash cartoon of the chef, or do they want to just see the menu? Palmer links interactivity to the theory of media richness, which has a whole body of research behind it that I am no expert on. But I would word my jury questionnaires to reflect a rating of appropriate interactivity.

The most important impact of this study is that it helps put usability studies on a more academically sound footing. It is very important to have evidence that you are measuring what you think you are measuring. It would be interesting to see if other studies have adopted these particular metrics because of the strong statistical evidence in this study.

The most straight-forward metric, download delay, is also one that has been discounted lately. The thought is that with so many users switching to broadband access, download speed is no longer the issue it used to be. This is especially false for sites with information seeking interfaces, which are often very dynamic and rely on database access. No amount of bandwidth will help if your site’s database server is overloaded.

Sphere: Related Content

Usability test of the Kent State IAKM home page

Thursday, December 11th, 2003

Note: this report shows the results of a usability test of the Information Architecture and Knowledge Management program web site at Kent State University in 2003. The site has since been redesigned.

1. Introduction

In usability study of the IAKM web site I found a number of serious problems. Current IAKM students were asked to complete a series of tasks using the site. Although participants were able to complete the tasks 91.67 percent of the time, they met all performance goals for each task only 36.11 percent of the time. The site is not fundamentally broken, but clearly there is room for improvement. Through statistical analysis, observations of the students, and remarks made by the students a number of issues were uncovered.

Many of the problems were global problems with site navigation and labeling, but there were also a number of prominent local problems. The severity of problems were rated via three categories:

  • Severe—prevents the user from completing a task or results in catastrophic loss of data or time.
  • Moderate—significantly hinders task completion but users can find a work-around.
  • Minor—irritating to the user but does not significantly hinder task completion. (Artim, 1).

Problems are also rated by scope. Any problem can be either global, meaning it applies to most pages or the site as a whole, or local, meaning it is particular to a page or specific section. Global problems are generally more pressing than local ones.

Findings are presented first in order of importance, followed by a description of the study methods.

  (more…)

Sphere: Related Content