Testing AI-Assisted research synthesis
UX CASE STUDY | DAMIEN LUTZ
Overview
About
Context:
A practical experiment, synthesising usability test data manually first, then running the same data through AI, to understand where AI adds real value in research synthesis and where a designer’s hand is still essential
My Role:
Designer, researcher, and problem-solver, with AI as a research synthesis collaborator.
Tech Stack:
Miro AI Synthesis
Introduction
I’d just completed moderated usability testing on a concept design for an internal Safety, Security & Wellbeing Platform—8 participants across Retail, 6 business units, each session an hour long, across 35 test scenarios.
After sessions, I had a large volume of qualitative data from myself and two co-facilitators to synthesise. I synthesised it manually first, then ran the same data through Miro’s AI synthesis tool to compare results.
A faster first pass on large datasets
AI can identify themes and crunch numbers across dozens of data points in seconds, freeing time for the deeper interpretation work only a designer can do
A useful cross-check on manual synthesis
Running AI and manual synthesis in parallel confirms where findings are solid and surfaces anything you might have missed or over-weighted
A tool that needs directing
AI synthesis is only as good as the structure and prompts you give it — knowing its limits is what makes it useful
What the AI did well
The AI’s insights were closely aligned with mine. It identified the same key themes and pain points, and it was particularly strong with numerical analysis—averaging ratings, calculating frequencies, and surfacing patterns across the dataset accurately and quickly.
For a dataset of this size, that kind of calculation would have taken considerably more time to do manually.
Correct themes
Identified the same key pain points and patterns as manual synthesis, with no prompting on what to look for
Correct calculations
Averaged ratings and calculated frequencies across 35 scenarios accurately and fast — the kind of task that eats time in manual synthesis
Where AI needed help
Accurate on themes and numbers, but the AI had clear limits. It stayed at surface level, missed context that was sitting right in the notes, and gave no indication of how many people were behind each finding. Left unchecked, those gaps can quietly skew your research.
Stayed on the surface
Identified themes but rarely dug into the detail behind them, even when that detail was in the data
No sense of scale
Reported what users said, but not how many — one person’s comment looked the same as eight people’s
Missed the emotional register
Patterns and pain points came through, but the texture of how participants felt didn’t
Four things I learned
Those gaps aren’t reasons to avoid AI synthesis—they’re reasons to know how to work with it. Running manual and AI synthesis in parallel surfaced four practical lessons for getting better output and knowing where to stay hands-on.
AI synthesis is fast, accurate on numbers, and good at pattern recognition across large datasets—a legitimate tool for research, but one that works best as a collaborator. The human work that remains: knowing what questions to ask, reading for emotional nuance, and making sure the numbers tell the whole story.
Prompt it to go deeper
The AI identified themes but missed the detail behind them, even when it was in the notes. Treat AI output as a starting point, then prompt for more
Tag before you synthesise
Tagging every note by role, team, scenario, and participant ID made it possible to ask for meaningful cuts of the data — without it, you get undifferentiated output
Always ask for the count
AI reported what users said without indicating how many. Always ask for the percentage or count behind each insight — it changes how you weight them
Read the raw notes yourself
If others are running sessions, read their raw notes before relying on AI output — you need to feel what participants experienced, not just read a summary