Usability Test Reporting

You know when a profession is mature, because the services and products offered by practitioners share a fair amount of consistency. So for example if I commission two different architects to carry out a house survey, their reports should be pretty similar. One may be cheaper than another, and one may be better able to describe the problems with the roof in terms I will understand, but the problems they find should be consistent.

Usability and variability

Embarrassingly, we have known for a while now that this doesn’t apply to usability testing. The well-publicised work of Rolf Molich shows us that when different usability groups are asked to carry out a web site evaluation, they find lots of usability issues. The problem is that each group finds only a sub-set of all the usability problems. Just one group (of nine) in Molich’s study found more than 25% of the problems. (More detail can be found at Molich’s web site).

Given that all these people would describe themselves as "usability professionals", it’s hard to blame the findings on different skill sets or competencies. A more likely contributor is the fact that the different groups carried out usability testing in a variety of different ways.

Usability standards

So it’s interesting that, during the period of Molich’s work, the US National Institute of Standards and Technology (NIST) initiated an effort to "Define and validate a Common Industry Format for reporting usability tests and their results". The overall aim of the project was to increase the visibility of software usability.

The Common Industry Format (or ‘CIF’ to its friends) isn’t a visual template that helps make usability reports look the same, nor does it tell you how to run a test. However, the framework of the report defines a consistent method of carrying out usability tests. For example, you can only write a compliant report if you take objective usability measures of effectiveness, efficiency and satisfaction (these definitions come from the international usability standard, ISO 9241-11). The report also requires information such as the design of the test (including information about independent variables), data scoring procedures (including operational definitions of usability measures) and details of the statistical analysis used. Following this type of guidance will help ensure consistency and contrasts with the more common approach, where usability tests aren’t "designed", they just happen. The CIF became an ANSI standard in December 2001 (ANSI/NCITS 354-2001) and became an international standard in 2006 ( ISO/ IEC 25062:2006 "Common Industry Format (CIF) for usability test reports").

Formative and Summative tests

The CIF makes a distinction between "formative" and "summative" usability tests. Formative tests are carried out:

• During the development of a product;
• To mould or improve the product;
• Virtually anywhere (you don’t need a lab);
• With the test administrator and the participant co-present.

The outputs from a formative test may include:

• Participant comments in the form of a "thinking aloud" narrative (for example, attitudes, sources of confusion, reasons for actions);
• Photographs and highlights videos;
• Usability problems and suggested fixes.

In contrast, summative tests are carried out:

• At the end of a development stage;
• To measure or validate the usability of a product;
• To answer the question: "How usable is this product";
• To compare against competitor products or usability metrics;
• To generate data to support marketing claims about usability;
• In a usability lab;
• With the participant working alone.

The outputs from a summative test may include:

• Statistical measures of usability (for example, success rate, average time to complete a task, number of assists);
• Reports or white papers.

The CIF applies to summative usability tests. The table below compares the advantages and disadvantages of the two methods.

Formative and Summative usability tests compared

Method	Advantages	Disadvantages
Formative or diagnostic test	Quickly highlights real problems. Verbal protocols valuable source of information. Can be used early in design to support rapid iterative development. Easy to prioritise problems.	Technique requires a test administrator who can keep the user talking. "Thinking aloud" can affect user behaviour and performance levels. Analysis of verbal protocols can be time consuming
Summative or measurement test	Provides real performance data. Answers the question: "How usable is this web site" Can compare different groups of users and different systems. High reliability and validity.	Technique requires a test administrator who knows how to avoid test bias. Technique requires a usability lab. Tasks can sometimes be artificial and restricted. Statistical analysis of data can be time consuming.

Achieving consistent tests

If your design process is human centred, and you aim to follow the process usability standard, ISO 13407, then you are probably carrying out summative usability tests already. The CIF will provide you with the consistent framework you need to report your results.

The usability tests in Molich's study used formative methods, to which the CIF doesn't strictly apply. So I hope Molich can be persuaded to repeat his study with summative methods and CIF-compliant reporting. This may show that the usability profession is more mature than we think.