Why Most Usability Tests Ask the Wrong Questions

You ran five usability tests. Participants stumbled on the checkout flow. You filed the findings. The team shipped a fix. Three months later, the same drop-off shows up in a different part of the product.

This is the most common failure mode in UX research — and it has nothing to do with sample size, recruitment, or tool selection. It starts with the questions you wrote before a single participant walked through the door.

After leading and consulting on user research and design strategy across enterprise SaaS, fintech, healthcare, and consumer products, I've seen the same pattern repeat: teams invest heavily in running usability studies, but underinvest in the quality of their research questions. The result is findings that describe what happened without explaining why — and recommendations that fix surfaces while the underlying problem quietly migrates.

The Observation Trap

Most usability test scripts are built around a task list. "Find the pricing page." "Add an item to your cart." "Complete the onboarding flow." The facilitator watches. The observer takes notes. Success or failure is recorded.

This is observation, not research.

Observation tells you that four out of five participants couldn't find the settings menu. Research tells you that users' mental model of "settings" doesn't match your information architecture — that they expect account-level controls to live inside the profile, not behind a gear icon in the nav bar.

The distinction matters because observation produces surface fixes (move the icon, add a label) while research produces structural insight (your navigation taxonomy conflicts with how users categorize their own actions). One fixes a screen. The other fixes a product.

What a weak research question looks like

• "Can users complete the signup flow?"
• "Do users notice the new feature banner?"
• "How long does it take to find X?"

These questions are answerable, but they're thin. They produce binary or numeric outputs — yes/no, 45 seconds, 3 out of 5. They don't generate the kind of insight that helps a product team make better decisions next time.

What a strong research question looks like

• "What expectations do first-time users bring to the signup experience, and where does the product violate those expectations?"
• "How do users decide what's relevant to them when encountering new functionality for the first time?"
• "What information do users need — and in what sequence — to feel confident completing this task?"

Strong research questions are generative. They don't just measure a current state; they reveal the cognitive structure underneath it. They give designers and product managers a transferable understanding they can apply to the next feature, the next flow, the next product decision — not just the screen being tested.

The Five Layers of User Struggle

When a user struggles in a usability test, there are five possible layers where the breakdown is occurring. Most teams only test for the first two.

Layer 1: Visual — "I can't see it."

The element isn't visible, is too small, has poor contrast, or is positioned outside the user's scan path. This is the easiest layer to test for and the easiest to fix.

Layer 2: Labeling — "I see it, but I don't know what it means."

The terminology doesn't match the user's vocabulary. "Workspace" vs. "Project." "Dashboard" vs. "Overview." This is where card sorting and tree testing add value, but most teams skip straight to prototype testing and miss it.

Layer 3: Conceptual — "I understand the words, but not why I'd use this."

The user can read the label and identify the element, but doesn't understand how it fits into their workflow. This is a mental-model mismatch, and it requires interview-style probing during the test to surface.

Layer 4: Motivational — "I understand it, but I don't care."

The feature solves a problem the user doesn't feel. This is a value proposition gap, not a usability issue — but it shows up in usability tests as hesitation, low engagement, or "I guess I'd click here?" language. Teams that don't listen for it misclassify motivation problems as design problems.

Layer 5: Trust — "I understand it and I want it, but I don't trust it enough to act."

The user understands the action and wants the outcome but hesitates because something about the experience doesn't inspire confidence. This is common in fintech, healthcare, and any domain where the stakes feel high. It requires testing for emotional response, not just task completion.

Designing a usability study that reaches all five layers means writing tasks that create space for Layer 3–5 struggles to surface, and writing a discussion guide with follow-up probes calibrated to each layer.

How to Restructure Your Research Questions

Here's a framework I use with UX research consulting clients to transform surface-level test plans into studies that produce actionable, structural insight.

Step 1: Start with the decision, not the interface

Before writing a single task, ask: "What product decision will this research inform?" If the answer is "whether the design works," you haven't gone deep enough. Push until you reach something like "whether we should restructure the onboarding around use-case selection or role-based paths."

A research study attached to a real decision automatically generates better questions, because the team needs explanatory insight, not just a pass/fail score.

Step 2: Map your assumptions

List every assumption the team is making about users in this area of the product. "Users understand what a 'workspace' is." "Users will want to invite teammates during onboarding." "Users will check notifications daily." These assumptions are your real test targets. Each one becomes a research question.

Step 3: Write questions at the mental-model level

Instead of "Can users find X?", ask "What do users expect to find in this part of the product, and how does that compare to what's actually there?" This reframes the study from validation to discovery. You'll still learn whether users can find X — but you'll also learn what they were looking for instead, which is almost always more valuable.

Step 4: Build in comparison moments

Ask participants to compare your product to their current workflow, a competitor, or their expectations. "How does this compare to how you do this today?" produces richer data than "Was this easy to use?" because it activates the user's real-world context instead of asking them to evaluate the product in a vacuum.

Step 5: Design for the follow-up question

The best facilitators spend 30% of a session on tasks and 70% on follow-up. Build your guide with explicit follow-up probes: "You hesitated there — what were you thinking?" "You said 'I guess' — what made you uncertain?" "If this were real money / real patient data / your actual work, would you have done the same thing?"

The follow-up is where Layer 3–5 insights live. If your test plan doesn't budget time for it, you'll only ever capture Layers 1 and 2.

The Real Cost of Thin Research

When usability testing only captures surface-level findings, the cost compounds across the organization in ways that are easy to miss:

Design teams iterate on the wrong things. They polish interfaces that have conceptual problems, shipping pixel-perfect screens that users still don't understand.

Product managers lose trust in research. When findings are obvious ("users couldn't find the button"), stakeholders start questioning whether research is worth the investment. The irony is that better research questions would produce the non-obvious insights that make research indispensable.

The same problems recur. Without structural insight, fixes are local. Move the button here, change the label there. The underlying mental-model mismatch stays, and surfaces again in the next feature, the next flow, the next quarter.

Research becomes reactive instead of strategic. Teams that only run surface-level usability tests position user research as a final checkpoint — a gate before launch. Strategic UX research and design strategy work should be shaping product direction, not just polishing what's already been decided.

What Changes When You Ask Better Questions

I've seen teams transform their product trajectory by changing nothing about their research operations except the quality of their questions.

One B2B SaaS team I consulted with had been running usability tests for two years with completion-rate metrics. They knew where users dropped off but had no idea why. By restructuring their research questions around mental models and decision-making patterns, they discovered that their users didn't think about the product in terms of "projects" at all — they thought in terms of "clients." That single insight restructured their entire information architecture and reduced support tickets by 40% in the following quarter.

Another team in healthcare technology was testing a patient-facing portal and consistently getting "easy to use" ratings. The problem was that patients weren't using it — adoption was flat. When we shifted the research questions from usability to trust (Layer 5), we discovered that patients didn't believe the portal was actually connected to their doctor. The interface was fine. The confidence gap was the real barrier.

These aren't usability findings. They're strategic product insights. And they came from the same method — usability testing — run with better questions.

A Note on AI and the Future of UX Research

With the rise of AI-powered UX research tools and automated usability testing platforms, the temptation to scale research through automation is real. These tools can capture clicks, heatmaps, and task-completion metrics at scale. But they can't ask "what were you thinking when you hesitated?" They can't probe Layer 4 motivation or Layer 5 trust.

As AI handles more of the observation layer, the value of a skilled UX researcher shifts entirely to question design, interpretation, and strategic framing. The researchers and UX consultants who thrive in this landscape won't be the ones who run the most tests — they'll be the ones who ask the questions no one else thought to ask.

← Back to all articles