We asked a representative sample of RBS employees to visit a web site that contained the intranet's top-level navigation terms arranged in a tree structure (this helped us focus on navigation without the distractions of aesthetics). The participants' task was to choose the right link for various tasks, such as "Find an expenses claim form". Over 200 participants took part in the study.
The challenge with a study like this is presenting the results back to the design team in such a way that they can make an informed decision on the data. There are some obvious statistics to use — such as the number of participants who succeeded in the task — but equally useful for design is an understanding of the incorrect paths chosen by participants.
The picture shows an example (for one task) of the way we chose to present the results.
Note the following features of the graphic:
- The tube map diagram shows the main paths participants took to find the answer. The green line shows the correct path and the red lines show commonly taken incorrect paths. A red circle indicates a node where people chose the wrong path.
- Success rate shows the percentage of participants who found the correct answer. The error bars show the 95% confidence interval.
- Success rate — detailed breakdown provides more background on the success rate measure, showing how many participants needed to backtrack to find the answer ("indirect success").
- Directness is the percentage of participants who didn't backtrack up the tree at any point during the task. The higher this score is, the more confident we can be that participants were sure of their answers (even if the answer is wrong). The error bars show the 95% confidence interval.
- Time taken shows the median time taken by participants. The error bar shows the upper quartile. You can think of time taken as a measure of hesitation when completing the task.
- We also included a qualitative judgement on how the design performed on this task based on the measured success rate ('Very poor' through to 'Excellent') and a section that interprets the findings and provides suggestions for improvement.
Other than the tube map visualisation, we were able to extract most of these metrics from the online tool we used to collect the data (Treejack). This made the analysis and presentation relatively straightforward.
(Many thanks to Rebecca Shipp, User Experience Manager, Digital Communications and Marketing at RBS Group, for collaborating with us on this project and allowing us to write about it).