Tag Archives: user satisfaction

Alexa business goals media richness metrics performance metrics project management reliability Usability usability testing user-centered design validity web designers

Notes: Web site usability, design, and performance metrics

Palmer, J.W. (2002). Web site usability, design, and performance metrics. Information Systems Research, 13(2), 151-167.

In this study Palmer looks at three different ways to measure web site design, usability and performance. Rather than testing specific sites or trying out specific design elements, this paper looks at the validity of the measurements themselves. Any metrics must exhibit at least construct validity and reliability—meaning that the metrics must measure what they say they measure, and they must continue to do so in other studies. Constructs measured included download delay, navigability, site content, interactivity, and responsiveness (to user questions). The key measures of the user’s success with the web site included frequency of use, user satisfaction, and intent to return. Three different methods were used: a jury; third-party rankings (via Alexa), and a software agent (WebL). The paper examine the results of three studies, one in 1997, on in 1999, and one in 2000, involving corporate web sites. The measures were found to be reliable, meaning jurors could answer a question the same way each time, and valid, in that different jurors and methods agreed on the answers to questions. In addition, the measures were found to be significant predictors of success.

This is an interesting article because in my experience, usability studies are often all over the place, with everything from cognitive psychology and physical ergonomics to studies of server logs to formal usability testing to “top ten usability tips” lists. Some of this can be attributed to the fact that it is a young field, and some of it is due to the different motive fueling research (commercial versus academic). One thing in the article I worry about, however, is any measure of “interactivity” as a whole. Interactivity is not a simple concept to control, and adding more interactivity is not always a good idea. Imagine a user trying to find the menu on a restaurant’s web site—do they want to be personally guided through it via an interactive Flash cartoon of the chef, or do they want to just see the menu? Palmer links interactivity to the theory of media richness, which has a whole body of research behind it that I am no expert on. But I would word my jury questionnaires to reflect a rating of appropriate interactivity.

The most important impact of this study is that it helps put usability studies on a more academically sound footing. It is very important to have evidence that you are measuring what you think you are measuring. It would be interesting to see if other studies have adopted these particular metrics because of the strong statistical evidence in this study.

The most straight-forward metric, download delay, is also one that has been discounted lately. The thought is that with so many users switching to broadband access, download speed is no longer the issue it used to be. This is especially false for sites with information seeking interfaces, which are often very dynamic and rely on database access. No amount of bandwidth will help if your site’s database server is overloaded.

How to design a really great website for your business

I’ve done a lot of small freelance web development projects and I find myself explaining my approach to non-technical managers, small business owners, and other decision makers again and again.  Here’s my attempt to write a more general document with a business perspective on web design.

Website design doesn’t just mean graphic design applied to a website.  There’s a lot of work that should go into the project before anyone starts designing anything, and I think a lot of companies and web designers skip these steps and end up with yet another site that does nothing for the company.

The first thing you’re going to want to do is determine your business goals for the site and your various user populations and their goals.  At this point you’re just brainstorming; later, we’ll figure out which goals are more important, which are most cost-effective, and figure out how your goals coincide with your users’ goals.

So let’s say, for example, that you have the following goals and user groups with their own motivations:

Business Goals:

  • Sign up new customers
  • Sign up new advertisers
  • Keep current customers
  • Reduce time/effort spent handling incoming files
  • Reduce time/effort spent handling ads
  • Communicate with customers
  • Sell additional services
  • Sell branded physical products
  • Reduce time spent on tech support

User groups/goals:
Current customers

  • Send in files for processing
  • Get help
  • Use your products to communicate with their customers
  • Have a web presence of their own
  • Improve their own business

Current advertisers

  • Send in material for ads
  • Make payments/view account status

Prospective customers

  • Find an affordable way to get their products produced
  • Help support their organization
  • Figure out which company to go with

Prospective advertisers

  • Find cost effective way to get more business
  • Figure out best place to advertise

As you can see, your goals do not exactly match up with your users’ goals, but there are many places where they overlap.  For example, your current customers want to send in their files, and you want to reduce time and effort handling incoming files and in tech support.  So that means the site should allow customers to send in their files as quickly and easily as possible.

In general, my rule is to look at the customers’ goals first, then match the business goals to them.  If you end up with business goals that do no fall under any of your users’ goals in the end, you need to think about how you will market this goal to users or attract user groups that share that goal.  For an (exaggerated) example, one of your business goals might be to sell an old backup generator.  Right now that meets none of your users’ goals, so unless you attract different users you won’t be any more likely to meet your goal than blind luck.

The next step is to prioritize and quantify these goals.  So perhaps we decide that we are going to get most new customers through face-to-face meetings with marketing reps and not through the web site.  That means we can bump up all the goals relating to current customers to the top of the list.  Or maybe attracting new advertisers is much more important than letting current advertisers pay online.  So we bump “sign up new advertisers” up the list.

In order to quantify the goals, we’ll need to establish some measures and current values for them.  So, for example, we want to quantify the “reduce tech support” goal.  How can we measure this?  Well, one measure is the number of people in tech support.  This is a bad measure, though, and a common mistake–if laying people off is your goal, that means you can make create a poor website and still lay people off and meet your goal.  Some better measures would be time spent per call, number of calls per customer, etc.  That way your employees can be more efficient and handle more customers or provide better quality help.  Also, always look at this from the customer’s point of view–why not measure how happy customers are with tech support overall, how happy they are with the help available (or not available) on the site, how likely they would be to use the site if they knew help was there, etc.

Once you have current figures for these measures, you’ll be able to tell if the site has really improved.  Once we know our customers’ goals and how our business goals match them, and we have figured out measures for each, we can start planning how to meet them.

This was just an illustration, but this is the kind of process that separates successful web sites from mediocre ones.  Keep in mind that this is just the first step toward a successful project, there’s still project management (planning the steps out and tracking milestones), picking the right technologies, designing the information architecture, testing usability, etc.

It’s a good idea to put together reports on each of these stages.  This is something a lot of web designers and companies don’t usually do, but I think it’s important for a project this size.  It keeps you aware of what’s going on, and eliminates duplicate work in the future.