What is benchmark usability testing?
Benchmark usability testing is very different from conventional usability testing. First, participants don't come to a lab instead, the participant carries out the test from his or her home or place of work. Second, a moderator does not interact with the test participant instead participants work with an automated system that asks them to perform tasks, measures their performance and asks for subjective comments.
"I like the dashboard approach and the tie in to ISO 9241 matches what I consider to be best practice (and we're an engineering company so standards are popular with my audience). For what it’s worth this is much better than the usual stuff that’s touted as a usability report and I’ll recommend you to contacts looking for similar services."
If you would like to see benchmark usability testing in action, you can take a tour from the perspective of a test participant. You can also view a sample report from a benchmark usability test (pdf format).
Why do benchmark usability testing?
Benchmark usability testing takes less time, has more reach and involves more participants than any other form of usability testing. It is a particularly good solution if you want to measure the usability of your web site or intranet rather than just find usability problems. It has the following benefits:
- We can collect robust usability metrics. Because we use a large number of participants in each evaluation (over 100), you can collect reliable measures of task completion rate, time on task and user satisfaction: the three cornerstones of usability as defined by the usability standard, ISO 9241.
- You can benchmark your site against the competition. We can direct half the participants to your site and the other half to your competitor's web site and then make a statistical comparison to show how you stack up. Or if you want to know if your design agency's proposed new site is really more usable than your existing web site, we can put them head-to-head. In fact, this method supports all variations of A/B testing.
- The testing set-up reflects real-world usage. Each participant uses the site alone for about 15 minutes, which is a better reflection of actual usage than a traditional usability test, where the participant is accompanied and may be asked to use the site for over an hour. And because participants carry out the test in their home or place of work, this kind of testing also has a more realistic contextual element. (If you like the contextual element but you're less interested in large sample sizes, try a remote usability test.)
- It's cheap. Because the test is unmoderated, we're able to price it much cheaper than a traditional, labour-intensive usability test. A standard test starts at £3,960 + VAT.
Case study: Customer experience benchmarking for Orange
The orange.co.uk web site is a critical channel for customers who wish to purchase talk plans, handsets and accessories. We were commissioned by Orange to help the company develop methods for specifying and measuring the usability of its web site so that it could be regularly benchmarked against the competition. Read usability benchmarking case study.
When should I do benchmark usability testing?
Benchmark usability testing requires a live web site or intranet or a stable electronic prototype that can be accessed via a web browser. Our system will work with all content, including Flash and Ajax. You don't need to add any code or make any changes to your web pages to use our system.
Our approach to benchmark usability testing
We define a number of critical tasks for your web site or intranet "red routes" and then arrange for participants to carry them out. Participants can be drawn from your own customer database or you can use our consumer research panel. Our panel comprises a demographically and geographically diverse set of users who have all signed up to take part in these evaluations. We precede each evaluation with a questionnaire, which means we're able to screen participants according to any criteria that matter to you: for example, if you just want to use Facebook users, we're able to meet that requirement.
We've developed our methodology to work on any platform and any browser, and participants don't need to install any software to carry out the evaluation. Because participants take the test from their own computer, we can test any web site that the user can access, including an intranet.
Participants are rewarded with a small incentive and usually complete the evaluation in around 15 mins.
What do I get?
Results are collected automatically by our web server and then analysed by our research team. We issue a report containing statistical measures of usability within 2 days of completing the test. See a sample report (pdf format).
We are experts in benchmark usability testing and in the statistical analysis of usability data.
User Experience Research
- Usability testing
- Field research and customer visits
- Usability expert review
- Remote usability testing
- Benchmark usability testing
- International usability testing