As a qualitative researcher, I’ve spent hours in the homes of the over-indebted, police officers, struggling minorities, victims of crime, successful executives, problem gamblers, budding artists and recovering alcoholics.
But looking back, the interview that taught me the most valuable lesson was with Charlie – a middle class mother-of-three from East London.
What people say
Charlie and I spent four hours in her kitchen, talking through her experiences and responsibilities, slowly building a picture of her life as a ‘busy mum’. Throughout, she depicted herself as a fierce fighter for social justice. She was proud to tell me that her friends knew her as ‘the political one’, ‘the big-mouthed lefty who is always informed’, ‘the one who others ask about what’s going on’.
This was ‘what Charlie said she did’. Or, more accurately, it was ‘what Charlie said other people said she did’.
But either way, it was just ‘stated behaviour’. Taken in isolation, I had no way of knowing if it was true. And yet, in the design sector, the misconception persists that stated behaviour is enough to establish what users want. The standard model for user research goes:
(1) Client asks what a far-away customer wants
(2) Researchers ask a variety of customers ‘what do you want?’
(3) Researchers package verbatim into a set of tidy quotes / stories, which are delivered back to the client
My research with Charlie helped me uncover what’s missing from this flow: the crucial ingredients of successful design research.
What people do
In this specific project, we were trying to understand how individuals use diverse sources to make decisions in their everyday lives. The method was forensic behavioural tracking: four-hour in-home ethnographic visits, coupled with online tracking (via browsing histories), video diaries, plus audits of app and social media usage.
By combining these sources over time, we revealed that, despite declaring herself an avid online Guardian-reader, Charlie never looked at news online. For the duration of the study, her browsing history consisted chiefly of holiday searches, online shopping and a few celebrity gossip articles.
Similarly, in trying to unpick why she’d voted for Jeremy Corbyn in the Labour leadership election, we established that she’d done very little research. Charlie didn’t even know the name of the alternative candidate, let alone any pledges made by either contender. Her social media bubble was filled with only one name, and that was the one she threw back into the world.
And yet Charlie still proudly described herself as ‘highly informed and curious’.
Why the gap?
Charlie’s case is far from unusual – and you can see why. For someone to explain ‘what’s really happening’ or, in the case of design research, ‘what they really want’, they’d need to be able to:
– Know what they’re currently doing (and by extension what they need/want)
– Know how to articulate it in a way that a researcher can understand
– Have to want to tell you
Getting all three at the same time is rare – so it’s just as well that, as researchers and designers, there are other things we can do to untangle the ‘real’ from the ‘stated’.
Context is king
The key is data triangulation – systematically gathering information from related data points (for example, other people, paperwork, photos, browsing histories, social media – even just observation). Supported by detailed, determined analysis, this enables us to ‘validate’ self-report data: identifying and explaining the consistencies and discrepancies that, together, signpost us towards ‘real’ behaviours, needs and opportunities.
This is why context matters so much. How else can you understand the experience of someone who is over-indebted without being in their derelict home, meeting their three children? How else can you map how gambling operators train their staff in social responsibility without being in the betting shop – seeing how the ‘official line’ plays out on the ground?
The truth is that relying simply on what people tell you will never tell you the whole story – and could even be actively unhelpful. To go back to Charlie for a second: she might say she wants an extensive news app, but our holistic research showed she probably wouldn’t use it.
Recent graduates from the Design Thinkers Academy – with whom we were collaborating to better embed user research in their own decision-making – discovered this for themselves in the Rheumatology Department at Guys and St Thomas’ Hospital. Interviewing patients in the waiting room, the data they collected was subject to a huge number of biases – all brought on by the context.
These were people quite literally at a ‘pain point’ in their journey. Many had been waiting a long time for an appointment, making them (understandably) more likely to complain about pain and overstate their waiting time. Others, meanwhile, thought that expressing negative feedback about their experience might have an effect on their spot in the queue.
Simply recording what they were told would have given the delegates a skewed impression of what mattered most to people – potentially leading to some less-than-effective recommendations about what needed to be done. But by understanding the context, and exploring it for themselves, they were able to interpret the patients’ testimony far more meaningfully – understanding and prioritising what really needed be done to improve the user experience.
Originally written for the Design Thinkers Academy.