You cannot see your own interface; you already know where everything is. Evaluation is the discipline of replacing that blind spot with evidence: planning what to learn, watching real people use the design, measuring what happened and why, and feeding the findings back into the next version. This companion ends with the concrete toolkit you need for an evaluation phase.
After weeks on a screen you are the worst possible judge of whether it is clear, because you already know where everything is. Evaluation is the discipline that replaces that blind spot with evidence from people who do not.
Every designer eventually has to face one question: is this any good? Your own judgement cannot answer it. You have built a mental model of the interface that no first-time user shares; the labels make sense to you, the flow is obvious to you, the error you never hit does not exist for you. This is not carelessness, it is unavoidable: knowledge of a design cannot be unlearned. The only way past it is to watch someone without that knowledge try to use the thing.
Evaluation is humility, made into a method.
The discipline in one lineSo evaluation is not an optional polish at the end. It is the activity that closes the design loop from the interaction-design chapter: discover, design, prototype, evaluate, and back again. Without it, a team ships its own assumptions and finds out whether they were right from users who have already left.
Two well-documented cases show what a single untested design decision can cost, in money and in worse.
Names ran down two facing columns with a single row of punch holes between them, so the second name on the left lined up with the third hole, because the top name on the right had claimed the second. A voter reaching for the second name punched the second hole and recorded a vote they never intended.
Over 19,000 ballots in the county were spoiled by double punching, against a certified statewide margin of 537. This was not a software bug; it was a mapping failure in a paper layout that no one tested with ordinary voters.
The butterfly ballot put two columns of candidates around one shared, staggered column of punch holes. The second name on the left did not line up with the second hole; it lined up with the third. Thousands of voters punched the wrong hole, and the layout plausibly changed the outcome of a national election.
No new technology was needed to prevent it, only a layout that mapped names to holes. The failure was pure interaction design, and it was never tested with voters.
A large retailer forced shoppers to register before they could buy. The checkout showed a Register button and a login form. In testing, first-time buyers were not confused, they were furious: they did not want a relationship, they wanted shoes.
Replacing the register step with a Continue as guest option reportedly produced around three hundred million dollars a year in extra revenue. The barrier was invisible to the team and obvious the moment someone watched a real buyer hit it.
The cases are vivid but old and small; the same lesson holds for large modern systems. The shared cause is always the same: a decision that looked fine to the people who made it, never put in front of the people who would live with it. The fix in every case was cheap, an afternoon with a few users in a quiet room, set against the cost of the failure.
Evaluation comes in two kinds, distinguished by when you do it and what you do with the result.
| Kind | When | Purpose |
|---|---|---|
| Formative | During design, on prototypes. | To improve the design while it can still change. The findings feed straight back into the next version. |
| Summative | After a version is built or shipped. | To judge how good it is, against a benchmark or a competitor. The findings grade the design rather than reshape it. |
A kitchen makes the distinction concrete. When the chef tastes the sauce while cooking and adjusts it, that is formative. When the guests taste the finished dish, that is summative. Either way, somebody has to taste it: looking at the pot is not evaluation. Most of what a project team does is formative, because the point is to improve the design while there is still time to act on what you learn.
The distinction is Michael Scriven's (1967), borrowed from educational evaluation: formative evaluation shapes something while it is being made; summative judges the finished result.
Evaluation is not "grab a friend and poke the app". It is planned. DECIDE, the framework of Rogers, Sharp and Preece, is six questions answered in order before you test anything.
What are the goals of this evaluation? What do you actually need to learn?
What specific questions will answer those goals?
Which approach and methods fit: lab, field, or inspection?
The practical issues: who, where, what equipment, how long, how many participants?
How will you handle ethics? Consent, privacy, and the right to stop.
Collect, analyse, interpret and present the data as it is.
Most weak evaluations skip the front and the back: they run sessions without first deciding what they need to learn (the first D and E), and they gather data without analysing what it shows (the last E). The two letters at the front protect you from the most expensive mistake, running a study that cannot answer your question. Answer all six, in order, and the rest of the study almost designs itself.
The "Choose" step picks from three families of method. Each answers a different question, and each is strongest at a different stage of the project.
Watch real people attempt real tasks, in a lab or a controlled setting. Answers: can people actually use it, and where do they struggle?
Observe use in its natural setting, over time. Answers: how does it fit into real life and work?
Experts judge the design against heuristics, or models such as Fitts's law predict performance, with no users. Answers: what is likely wrong before we recruit anyone?
The families are not rivals; they attach at different points, scaling from no users to many as the design firms up.
| Stage | Methods | Users |
|---|---|---|
| Discover | Interviews, personas, scenarios | a few, for context |
| Design | Heuristic inspection, Fitts's law models | no users |
| Prototype | Usability testing | a few users |
| Live | Analytics, A/B testing | many users |
The cheapest family is heuristic evaluation, introduced by Jakob Nielsen and Rolf Molich in 1990: several evaluators independently judge the interface against a short list of recognised principles, then pool their findings. Three to five evaluators usually uncover most of the problems an inspection can find, because no single evaluator catches everything. Nielsen's ten heuristics are the standard checklist.
| Heuristic | What it checks |
|---|---|
| 1. Visibility of system status | The system keeps the user informed about what is going on, through timely feedback. |
| 2. Match with the real world | It speaks the user's language and follows real-world conventions, not system jargon. |
| 3. User control and freedom | It offers a clearly marked way out: undo and redo, an emergency exit from a wrong state. |
| 4. Consistency and standards | The same words and actions mean the same thing throughout, and follow platform convention. |
| 5. Error prevention | It designs out the error in the first place, better than any good error message. |
| 6. Recognition rather than recall | Options and information are visible, so the user need not remember things between steps. |
| 7. Flexibility and efficiency | Accelerators (shortcuts, defaults) speed the expert without obstructing the novice. |
| 8. Aesthetic and minimalist design | It shows only what is relevant; every extra unit competes with the important ones. |
| 9. Recover from errors | Error messages are in plain language, state the problem, and suggest a fix. |
| 10. Help and documentation | Where help is needed it is searchable, task-focused and concrete. |
These are the same family as Norman's principles from the interaction-design chapter, turned into an inspection checklist. Heuristic evaluation needs no users, so it is run early to clear the obvious faults before a participant is ever recruited.
A list of problems is not yet actionable; each needs a severity so the team fixes the worst first. Nielsen's severity scale folds frequency, impact and persistence into a single rating from 0 to 4.
| Rating | Meaning |
|---|---|
| 0 | Not a usability problem at all. |
| 1 | Cosmetic: fix only if spare time allows. |
| 2 | Minor: low priority. |
| 3 | Major: important to fix, high priority. |
| 4 | Catastrophe: imperative to fix before release. |
The findings table at the end of this chapter carries exactly this rating, so the revision plan is ordered by severity rather than by whoever argued hardest.
Beginners treat evaluation as one event: run a test. It is really a chain of six linked steps, and a weak link anywhere wastes the rest.
What this round must learn, taken straight from DECIDE.
Real tasks that exercise the goal, framed as outcomes not instructions.
The run itself: a participant, the tasks, think-aloud, and a quiet facilitator.
What you record: completion, time, errors, and the reasons behind them.
The problems found, ranked by how badly they hurt the task.
The changes you will make, and the loop back to a fresh goal.
The chain ends where it began: a revision sets the goal for the next round. That loop is exactly what a project's evaluation phase has to show, not a single test in isolation but evidence that findings led to changes, and that the changes were checked.
Strip away the equipment and a usability test is one person trying to do a real task while you watch and stay quiet. The structure is what turns watching into evidence.
A synthetic illustration, not a recording of a real person. The task succeeds, yet the two wrong turns and the time taken are the evidence: you would not ship a menu nobody can find. Friction is a finding even when the task is completed.
The common worry is that a handful of participants cannot be representative. For finding usability problems, that worry is misplaced. Nielsen and Landauer (1993) modelled the number of problems found as N(1 − (1 − L)n), where n is the number of users and L is the proportion of problems a single user finds, on average about 0.31. With L = 0.31, five users surface roughly 85% of the problems, because the serious ones are hit by almost everyone; Robert Virzi had reported the same pattern independently in 1992. After five, you mostly watch the same issues recur. The efficient move is not one big study but several small rounds: test five, fix, test five again. Summative studies that need a stable number, such as a benchmark score, require more participants, but for finding what to fix, five per round is the working rule.
The curve is steepest at the start: the first user alone finds about a third of the problems, and by the fifth the line has all but flattened. Spending the sixth, seventh and eighth participant on the same round mostly re-finds what you already know; a fresh round of five after a fix is the better use of them.
A task must be framed as an outcome the participant wants, not a list of steps to follow. "Click Menu, then Settings, then Notifications, then toggle the switch" tests whether they can read; it cannot fail. "You want the app to stop sending you alerts at night, make that happen" tests the design, because now the participant has to find the path themselves, and where they go wrong is exactly the finding. Give the goal and the starting point, never the route.
Ask participants to say what they are thinking as they work. Clicks tell you what happened; words tell you why. "I am looking for a settings cog, usually it is top right" reveals an expectation the design can meet or break, which no log file would ever show. Thinking aloud is the cheapest way to recover the reasoning behind a wrong turn. The technique is verbal protocol analysis, formalised by Ericsson and Simon (1984): a concurrent protocol, narrated during the task, captures reasoning in the moment, while a retrospective one, gathered just after, disturbs the task less but loses detail to memory.
The hardest part of running a test is not helping. The moment you say "try the menu", the finding is gone: you have shown them the thing the design failed to. Let the silence sit, let the participant struggle, and record where the struggle happens. A facilitator gives the task, the encouragement to keep narrating, and nothing else.
The SUS is ten statements, each rated 1 to 5, that produce a single satisfaction score from 0 to 100. Published by John Brooke in 1996, it has become the most widely used standardised usability questionnaire, valued because it is short, free, and reliable even with the small samples a project can afford.
The wording alternates on purpose: odd-numbered statements are positive and even-numbered ones are negative, so a high rating means something different on each. That is what stops a participant from ticking the same box down the page without reading. It also means you cannot simply add the answers; every one first has to be converted to the same 0-to-4 scale where higher is always better.
Convert each answer, then sum and scale:
Worked example: take one participant's ten answers, convert each with the rule above, and suppose the converted scores add up to 29. Then 29 × 2.5 = 72.5. The benchmark to remember is 68: that is the average SUS score across hundreds of studies, so 72.5 sits just above average. SUS is a measure of perceived usability, not a percentage, so treat it as a relative score against that 68 line and against your own previous rounds. Jeff Sauro's analysis of hundreds of studies maps the raw score onto a percentile and a letter grade: 68 is roughly the 50th percentile (a C), and a score must reach about 80 to land in the top tier (an A). Bangor and colleagues add an adjective reading, where the low 70s correspond to "good" and the mid 80s to "excellent".
Beyond the moderated test, three methods earn their place: two that work at the scale of live traffic, and one that predicts performance with no users at all.
An A/B test splits live users between two versions and measures which does better on one metric. It is the most decisive method there is for the question it answers, because real users vote with real behaviour at scale. Its famous limit is that it can only choose between the options you give it. It will tell you that button B beats button A, but never that a different design C, which you did not build, would beat both. A/B testing optimises within the current idea; it cannot find a better idea, and it cannot tell you why the winner won.
Analytics record what many users do in the live product: which screens they reach, where they drop off, how long they take. The strength is scale, thousands of real sessions. The weakness is the same as A/B testing: analytics tell you what happened but never why. A cliff in the funnel shows where people leave, not what confused them. The why has to come from watching.
Fitts's law states that the time to move to a target depends on its distance and its size: the closer and larger the target, the faster it is to hit. It is a model, so it predicts performance from the design alone, before anyone is recruited.
Drag the sliders. Difficulty is the index of difficulty in bits, the bracket in the formula; the predicted time uses simple constants for a and b, so the exact milliseconds are illustrative and the relationship is the point. Double the distance, or halve the width, and the move gets exactly one bit harder; double both and the two effects cancel.
Because the law uses target width, a target pinned to an edge behaves as if it were infinitely deep: the cursor cannot overshoot the screen edge, so you flick towards it without aiming. A corner is infinite in two directions, the single easiest region on the screen, which is why menu bars sit at the top edge and why corners hold the most-used controls. Click the two targets below and feel the difference the law predicts.
last move: tap one target, then the other.
A usability session puts a person in a position where they may feel tested and may struggle in front of you. That carries obligations, and they are not optional.
The participant is doing you a favour by exposing your design's weaknesses. Reassure them often that any difficulty is the design's fault, not theirs; an anxious participant changes their behaviour, and you lose the very honesty you came for.
No single method sees the whole picture. Each has a blind spot that another method covers, so strong evaluation combines them rather than trusting one.
| Method | Sees | Is blind to |
|---|---|---|
| Inspection (heuristics, models) | Likely problems, fast and cheap, with no users. | The surprises only real people produce. |
| Usability testing | Real users, the why behind a struggle, the surprises. | Scale; a small sample in a lab, not life. |
| Analytics and A/B | What thousands of real users do, at scale. | The why; and any option you did not build. |
Run together, they cover each other: inspection finds the obvious faults cheaply before you spend a participant on them; testing explains the why that analytics can only point at; analytics confirms at scale what a handful of sessions suggested. The one rule that beats all method debates: watching one user beats watching none. A single session, run with care, will tell you more than another week of arguing about the design in a quiet room.
Pulling the chapter together: an evaluation phase is not one test but a small, complete kit of artefacts. Here is each one, ready to fill in.
One short page: the goal of this round, the questions it answers, the method chosen, the practical details, and the ethics. Written before anything else.
Five people who resemble the real users, not classmates who built the thing. Booked for about fifteen minutes each.
What the session involves, what is recorded, how data is stored and anonymised, and the right to stop. Signed before you begin.
Every problem, ranked by severity, with the evidence behind it, leading to a concrete change and a plan to re-test.
| Problem | Severity | Evidence | Change |
|---|---|---|---|
| Users could not find the night-mute setting | High | 4 of 5 stalled; "I expected a cog top right" | Move settings to a top-right cog; re-test |
| Term defaulted to last year | Medium | 2 of 5 registered for the wrong term | Default to the current term |
| Confirmation wording unclear | Low | 1 of 5 asked "did that work?" | Reword the confirmation; lower priority |
Goal: can a first-time user register for a seminar without help? Tasks: three goal-framed tasks. Session: five participants, about fifteen minutes each, thinking aloud, facilitator silent. Ethics: consent form, data anonymised, right to stop. Measures: completion rate, time, the points where people stalled, and a SUS score each. Findings and revise: the table above, then the changes you will make and the re-test that closes the loop. That loop, evidence leading to change, is exactly what an evaluation phase has to show.
The works this chapter draws on for the framework, the methods, and the cases.