The Data Says We're Doing Fine, So Why Are We Furious?

When Data and Anecdote Conflict

Photo Jack Taylor/The Times

Catch-up service:
How Humans Got Language
10 Classical Music Pieces To Take On Holiday
The Diderot Effect
27 Notes on Growing Old(er)
Donald Trump and the End of Power
5 Reasons There Won’t Be an AI Jobs Apocalypse

Jeff Bezos sat in a meeting with his senior leadership team, listening to the scratchy, looped music of Amazon’s customer service hotline for ten long minutes.

This was during the very early days of Amazon. His team were getting a lot of angry complaints from customers who had called the service hotline and been made to wait for ages. But the head of customer service insisted, week after week, that the hotline was working fine: the data showed most customers were getting through within sixty seconds.

Bezos was sceptical. So when the executive presented this data again at the weekly business review, Bezos suggested they call the hotline there and then. They did, and then waited in silence while nobody picked up. It must have been excruciating (I’m feeling bad for the executive just typing about it). But it prompted Amazon to investigate more deeply, and on doing so they found that the problem was real - they just hadn’t been measuring the right thing.

The experience led Bezos to coin an aphorism: “When the data and the anecdotes disagree, the anecdotes are usually right.”

This runs counter to conventional wisdom among those who consider themselves hard-headed realists, whether finance directors or political analysts. We’re meant to put our faith in the data, and ignore those silly, soft-headed anecdotes. And yet one of the most famously hard-headed - in every sense - men in business takes them very seriously. Jeff Bezos built Amazon on data, but he doesn’t entirely trust it.

In a podcast interview, Bezos explains that his aphorism doesn’t mean anecdotes should always trump data.¹ It means that when you have customers complaining about something your metrics say they shouldn’t be complaining about, you should question the metrics before dismissing the complaints. It’s not that the data is badly collected or poorly analysed, it’s that it’s directing your attention to the wrong problem.

I thought about this while reading John Burn-Murdoch’s typically excellent analysis of the politics of crime in America and Britain. Burn-Murdoch presents a paradox. We are in the middle of an inverse crime wave: this year is on course to show the lowest rate of violent crime in decades, in the US and the UK.

In a world where so many trends are moving in the wrong direction, this is a rare and important good news story. But are voters rejoicing? Are we congratulating our politicians, police forces, and bureaucrats on jobs well done? No, of course not.

In Britain, concern about crime has been trending upwards for several years, even in years when crime drops, and the same is true in the US. Populist politicians have been quick to make hay. Nigel Farage, leader of Reform, currently Britain’s most popular party, is warning of societal collapse. Donald Trump made crime a central plank of his successful election campaign.

Why on earth is crime such a hot political issue when the data shows it to be historically low levels?

After the jump, the real cause of crime concerns and what it means for how to make good judgements when data and anecdote are in conflict. PLUS a glorious Rattle Bag of links and notes on what I’ve been reading, viewing, and watching (including thoughts on David Hockney, Henry James, W.H. Auden, and Lucy Letby). If you haven’t already done so, please consider a paid subscription. Paid subscribers make The Ruffian possible.