Zeno's Paradox
While critical thinking may not make up for a lack of knowledge, it is essential for gaining knowledge.
Tuesday, December 17, 2002:  

Human Factors:

Professional Usability Testing and Return On Investment (57-page PDF file, 340kb) Summary (Charles L. Mauro - TaskZ)

After going through this report in detail, I highly recommend it for both its discussion of the return on investment of usability testing and its comparison of lab-based versus online usability testing. A few highlights with comments:
Definition: Professional usability testing is defined as a formal research methodology that adheres to the processes and rules of scientific investigation as developed and taught in formal graduate level programs in the cognitive sciences. Practitioners of this type of research hold advanced degrees in human factors engineering, ergonomics, or other relevant cognitive science fields.
This is what every usability roi argument requires, but few provide - a definition that qualifies what types of usability testing are being discussed and who must conduct the tests to produce the level of quality required. ROI arguments without such definitions are meaningless because of the wide variety of usability testing methods and the diversity of expertise of usability practitioners.
Until recently applying rigorous, professionally executed usability research in the design of web-based products and services has not been a part of most web development efforts. Generation One of the internet used an unstructured software development methodology that did not take advantage of professional usability testing.
Why has it taken so long for professional usability testing to be applied to web development efforts? Mauro in part blames the gurus:
Along with the dramatic increase in funding for internet startups during the late 1990s came the rise of guru usability experts offering services at very high rates. Along with the rise in guru usability came the concept that usability science could be force fed to development teams based on a few days of expensive consulting. This practice led to the mistaken impression that formal usability engineering was a quick fix, even for complex usability problems. This unfortunate trend led many in the software engineering community to view this important new science as transitory and having little effect. In reality, professional usability engineering and testing is a fully bona fide profession with a long and clear history for improving the design and acceptability of screen-based systems. Most of these gurus have migrated back to seminars where their views make sense and can be taken in context of the seminar setting. Guru usability does not have a place in complex product development settings.
I'm not sure that these gurus' views make sense anywhere. Perhaps Mauro is just being kind in what is otherwise a very damning commentary on what the gurus accomplished.

In Part 2 of the report, "Business Rationale Behind Professional Usability Testing," Mauro gives five "critical statistics" for usability testing:
1. For every dollar spent acquiring a customer you will spend $100 dollars reacquiring them after they leave because of poor usability or bad customer service.

2. More than 95% of your customers will use less than 5% of the features and functions of your site. Customers will NEVER use about 75% of the functions on your site.

3. The single largest predictor of call center volume is your web site’s usability. Calls cost an average of $22-$30 per call.

4. For every $10 spent defining and solving critical usability problems early in development using professional usability research, you will save about $100 in development costs.

5. For every dollar you spend improving the visual design or style of your site, you will receive virtually no improvement in sales. The same dollar spent on improving core behavioral interactions with your site’s critical way-finding and form-filling functions will, however, return $50-100 if executed in a professional and rigorous manner.
I'm always skeptical of such statistics, even though these are fairly well founded. Mauro qualifies them to a certain degree, but still leaves plenty of room for them to be misinterpreted.

Mauro also comments on some of the politics of usability testing, including:
It has always been a curious fact that some large E-Com clients will use rigorous usability testing and advanced human factors engineering methods at various phases of development, then ignore the results.

Before discussing online testing tools, Mauro discusses key aspects of successful testing:
In all successful professional usability testing programs, the following four key aspects of the formal research methodology must be present, properly linked, and managed.
Component 1: Clearly articulated hypothesis.
Component 2: Proper experimental design.
Component 3: Reliable data.
Component 4: Data must be properly interpreted.
Most usability practitioners can learn a great deal from Mauro's discussion on each component, including the gurus:
There has been a popular notion that conducting usability research with a small group of users can yield answers to complex usability questions. This approach popularized under the general description Discount Usability Testing is not an acceptable method for producing reliable data in an experimental setting. Certainly, we would never make mission-critical decisions using small subject samples with poorly articulated user profiles.
Not a complete dismissal of the most recent round of arguments on the number of test participants, but close...

Mauro goes into detail on the difficulties of usability testing. Again providing information that most can learn a great deal from:
Four paradoxes of usability research
Paradox 1: Objective and subjective data can be negatively correlated.
Paradox 2: What looks easy on first impression can be difficult to use.
Paradox 3: Subjective ratings can be skewed positive.
Paradox 4: Behavior is more important than look and feel.

As for online usability testing:
A review of the current online usability testing offerings shows that about 95% of the current systems are objectively little more than web-based survey systems.

Summary: Mauro has written a highly informative report that has a great deal high-quality information. Unfortunately, his solution of rigorous usability testing conducted by highly educated usability professionals will appear impractical and dismissive of common usability practices and practitioners.

    -  Ron  1:53 PM

Copyright © 2002-2005 Ron Zeno      This page is powered by Blogger.
Musings not completely unrelated to human factors, management, critical thinking, medicine, software engineering, science, or the like.
Zeno's Paradox Main
Comments? Email me!
My other site
Reader Favorites:
Miller's 7+-2 Doesn't Apply
Analysis of a Dilbert Comic
Reliable & Valid Usability?
Designing for Seniors

Weekly Archives
Looking for Ancient Greek Philosophy or Mathematics?
Zeno's Paradox
Zeno and the Paradox of Motion

(Current color scheme is from Aguilar's "Equilibrium".)