Usability testing, the methods around it, and a couple of laws that let you judge a design before anyone touches it.
The problem with your own design
Once you know where the button is, you can never again experience not knowing. This is the curse of knowledge, and it is why self review fails.
Knows the menu structure, the labels, the hidden gesture. Everything is obvious. Of course it is.
Sees a wall of choices, no memory of your intentions, and a goal that has nothing to do with your architecture.
Evaluation exists to measure the distance between those two people. You cannot guess it. You have to look.
You are not the user. You will never be the user again. Test with someone who is.
The certified statewide margin was 537 votes. This was not a software bug, it was a mapping failure in a paper layout that no one tested with ordinary voters. Recall Norman: when action and result do not line up, people make errors that are not their fault.
Nobody on the team had predicted this. They were too close to it. A handful of real sessions found money that years of internal debate had missed.
The same root cause, decades apart
Both systems were shipped without watching real people under real conditions. Both broke down in public, at the worst possible moment.
A control panel showed that a relief valve had been commanded shut, not that it was physically stuck open. Operators trusted the light while coolant drained away. Meanwhile hundreds of alarms lit at once with no priority. A feedback and mapping failure in a nuclear control room.
The United States health insurance exchange launched with almost no end to end usability or load testing. On the first day a mere handful of people managed to enrol before the site collapsed under demand. Evaluation skipped at the scale where it mattered most.
Different decades, different domains, one lesson. A system never tested with real people under real load will still find its problems. It just finds them in front of the public instead of in a quiet room with five users.
Before you test anything
Evaluation is not "grab a friend and poke the app". It is planned. Six questions, in order.
What are the goals of this evaluation? What do you actually need to learn?
What specific questions will answer those goals?
Which approach and methods fit? Lab, field, or inspection?
Practical issues: who, where, what kit, how long, how many?
How will you handle ethics? Consent, privacy, the right to stop.
Collect, analyse, interpret and present the data honestly.
Most weak evaluations skip D and E at the front. They run sessions without deciding what a useful answer would even look like, then drown in notes.
The map of methods
Set tasks, watch closely, in a quiet room. You control the conditions.
Precise, slightly artificial.Watch real use in the real setting, on a bus, in a shop, at 2am.
Realistic, hard to control.Experts judge against principles. Or you predict performance with a formula.
Fast and cheap, blind to surprises.We spend most time on usability testing, because it is the workhorse. Then we end with the without users methods, because being able to judge a design with no people in the room is a quiet superpower.
Where these tasks come from, and where they lead
Each step exists because the one before it produced something the next step needs. Nothing here is busywork.
DECIDE: what must we learn?
Turn the goal into real user goals
Observe, think aloud, stay quiet
Completion, time, errors, SUS
Themes and severity ratings
Fix, then test the fix
The workhorse method
Put a representative person in front of the real thing, give them a real goal, and watch without helping.
You are not selling the design and you are not defending it. You are collecting evidence about it, and the most useful evidence is usually the part that stings.
The 85 per cent assumes each user finds about a third of the problems, which holds for a focused interface but not for a sprawling, varied one. To measure a number precisely you also need far more people. The argument over this single figure has run for thirty years.
The skill nobody practises
The wording of a task decides whether you learn anything. Tell people what to achieve, never how.
"Click the Registration menu at the top, choose Add Course, then type COMP322 in the search box."
You just tested whether they can follow instructions. Useless."It is registration week. Add COMP322 to your schedule for a section that does not clash with your morning lab."
Now you see where the real journey breaks.A good task has a context and a clear finish line, so both you and the participant know when they are done. It should be something a real person would actually want to do.
Hearing the reasoning, not just the clicks
Ask participants to say what they are thinking as they work. Clicks tell you what happened. Words tell you why.
"I am looking for register... maybe under this menu... no, that is settings... wait, why is there a second login box... I will just try this one."
Nobody narrates their own thoughts in daily life, so people forget. Your job is to keep the stream running without putting words in their mouth.
The hardest job in the room
Every word you add contaminates the data. The instinct to rescue a struggling user is the instinct you must defeat.
"What would you do here?" beats "would you click that button?". The second hands them the answer.
If they are stuck, that is a finding. Wait. Let the silence stretch. The struggle is the result you came for.
No wincing when they miss it, no smiling when they get it. Your face is feedback too.
People want to please you. They will praise a design to your face and abandon it the moment you leave. Neutral questions and a neutral face are how you get the truth instead of the compliment.
Watch one session
Numbers and stories together
Quantitative data tells you that something is wrong. Qualitative data tells you why. You need both, or you are guessing.
A completion rate with no quotes is a thermometer with no diagnosis. You know there is a fever. You have no idea what to treat.
One number for satisfaction
Ten statements, each rated 1 to 5. The wording alternates on purpose, odd items positive and even items negative, so agreeing means something different each time. Before you can add the answers, every one has to land on the same 0 to 4 scale where higher always means better.
"I would like to use this often." Agreeing is good, so the top answer 5 must become the best score. Subtract 1, and raw 1 to 5 becomes 0 to 4.
"I found it unnecessarily complex." Agreeing is bad, so the top answer 5 must become the worst score. Subtract from 5, and the scale flips, 1 to 5 becomes 4 to 0.
| Raw answer | Odd: raw − 1 | Even: 5 − raw |
|---|---|---|
| 1 | 0 | 4 |
| 2 | 1 | 3 |
| 3 | 2 | 2 |
| 4 | 3 | 1 |
| 5 | 4 | 0 |
Odd, minus one. Even, from five. Subtracting from 5 reverses the scale, so disagreeing with a negative statement correctly counts as a good thing rather than a bad one.
From ten answers to one number
One participant's ten answers, converted item by item with the two rules, then summed and scaled. Synthetic example, participant P2.
| Item | Raw | Rule | Score |
|---|---|---|---|
| 1Would use it often | 4 | 4 − 1 | 3 |
| 2Too complex | 2 | 5 − 2 | 3 |
| 3Easy to use | 4 | 4 − 1 | 3 |
| 4Needs tech help | 1 | 5 − 1 | 4 |
| 5Well integrated | 4 | 4 − 1 | 3 |
| Item | Raw | Rule | Score |
|---|---|---|---|
| 6Too inconsistent | 2 | 5 − 2 | 3 |
| 7Quick to learn | 3 | 3 − 1 | 2 |
| 8Cumbersome | 2 | 5 − 2 | 3 |
| 9Felt confident | 4 | 4 − 1 | 3 |
| 10Lots to learn first | 3 | 5 − 3 | 2 |
Show version A to half your users and version B to the other half, then measure which performs better. At the scale of millions you can detect tiny effects, but you only learn which option wins, never why.
The team tested splash page variants. A "Learn more" button paired with a family photo beat the original by about 40 per cent more sign ups, an estimated 2.8 million extra subscribers and tens of millions in donations.
A debate over a link colour was settled by testing 41 shades of blue. A senior designer resigned, arguing that data had replaced design judgement entirely.
A/B testing picks the better of two options you already have. It cannot tell you that a third, better idea exists, and it cannot explain behaviour. It climbs the hill you are standing on, never asking whether you are on the right hill.
Drag the sliders and the predicted time and the cursor obey the law. Pointing time depends on just how far the target is and how wide it is.
Fitts's law, applied
Click the two targets in turn. Watch which one your hand finds faster.
Put your most important, most frequent controls at edges and corners, or right under the cursor. The law was published in 1954 and your operating system is quietly obeying it right now.
Non negotiable
Every method in this lecture involves a real human giving you their time and sometimes their frustration. Respect comes first, data second.
Open every session with it: "We are testing the design, not you. If something is confusing, that is the design's fault, not yours." It relaxes the participant and gives you honest behaviour.
No method is enough alone
Heuristic evaluation, fast and cheap, no users needed.
Misses the surprises only real people produce.Real users, real tasks, the unexpected made visible.
Slow, small samples, says little about scale.Millions of data points, or a formula with none.
Tells you what, almost never why.Inspect early to catch the obvious for free. Test with five users to find the real surprises. Use analytics to confirm at scale. Each method covers another's blind spot. That is triangulation, and it is what separates a real evaluation from a vibe check.
The lower bar you should clear today
The teams that improve fastest are not the ones with the best first design. They are the ones that put their work in front of a real person soonest and were willing to be wrong.
Where this lands for you
Everything in this lecture is the toolkit for the Phase 3 deliverable, due 24 June. Match each rubric line to a method you now have.
Phase 3 is not about proving your design was perfect. It is about showing you can close the loop: test honestly, find the problems, and say clearly what you would change next. Iteration is the grade.
You cannot see your own design. So you plan a question, watch a real person, measure what happened and why, and let the evidence change your mind.
Data gathering in depth: interviews, observation, and questionnaires done properly, the raw material every method here depends on.