Usability Evaluation Considered Harmful

// Comments

Brad Twitty: ... Considered Harmful

Brian S: Usability Evaluation Considered Harmful

J Chris: Usability Evaluation Considered Harmful

Usability Evaluation Considered Harmful (Some of the Time)

Saul Greenburg, Bill Buxton

Usability evaluation is a very important step in the design process, but it should be taken moderately. As more emphasis is being placed on usability design, though, it starts to over step its bounds. Applying usability evaluation to places where it should not be utilized can result in "meaningless or trivial results and can misdirect or even quash future design directions."

As usability evaluation is becoming more popular with conferences, people are trying harder and harder to incorporate it into their paper, even when another method would present more compelling information. People would rather take the time to do an extensive proof than take the time to conduct risky hypothesis testing that might result in a failure of their research. Most researchers would rather take the easy way out and create an environment that is favorable to their study than attempt to see where it fails and where it can be improved upon. This ultimately results a lack of replication of the study meaning the researchers results are the only ones that are presented.

More researchers want to conduct their usability evaluation as early and often as possibly as well. The problem is that this results in an evaluation on early primitive sketches of designs rather than a working prototype. This can be useful if taken into consideration but not taken as the final evaluation of the design. Ultimately, the goal is to conduct several small usability evaluations throughout the design process that will lead to a final working design.

Several steps can be taken to solve the usability evaluation problem. Researchers need to understand that usability evaluation is just one of many steps in the design process. They need to judge when a usability evaluation would actually produce meaningful results. They need to stop using usability evaluation as the only evaluation on everything, even when it should not be applied. When usability evaluations are useful, they should be conducted in such a way that they produce strong results. Finally, we can look at other disciplines as examples on how to judge the worthiness of our own designs.

I found this paper to be very informative into the design process. Though I have not attended any conferences, from the papers that I have read and were presented, I do not feel the situation is as grim as they present. I will admit there have been some evaluations that I feel would justify this paper, but not enough to warrant any kind of complete overhaul of the design process. There are some people doing it wrong, but there are enough people doing it, at least somewhat, correctly that all hope is not lost. I will say, though, that CHI and UIST will accept almost any paper based solely on their writing, not their actual research. They seem to have a bit of a bias in that area.