How to spot a suspicious statistic
08:58 Share

This BBC podcast is supported by ads outside the UK.

Welcome to It Takes Energy, presented by Energy Transfer, where we talk all things oil and natural gas. Oil and gas drive our economy, ensure our country's security, and open pathways to brighter futures. What do you know about oil and natural gas? You likely associate them with running your car or heating your home. But these two natural resources fuel so much more than that. More than 6,000 consumer products that we rely on every day are made using oil and gas.

Before you even step out the door in the morning, you've already used more products made possible because of oil and gas than you realize. From the toothpaste you brush your teeth with, the soap you washed your face with, and the sheets you slept on. Not to mention your makeup, contact lenses, clothes, and shoes. Oil and gas are vital parts of all these products and so many more.

Look around and you'll see the essential role oil and gas plays in our lives. Our world needs oil and gas, and people rely on us to deliver it. To learn more, visit energytransfer.com. Hello, and thanks for downloading the More or Less podcast. We look at the numbers and statistics in the news and in life. And I'm Tim Harford. Untruths sneak into our lives in all kinds of ways. Sometimes they're outright lies, blatant misinformation,

But today, we're going to talk about something else. Those sneaky numbers and claims that bounce around our society that aren't exactly false, but are leading you down the wrong path. That's the subject of a book called May Contain Lies by Alex Edmonds, a professor of finance at London Business School.

Let's start with a straightforwardly good piece of advice from Alex, if I do say so myself. If you see a result that you want to be true, then as you say in your own book, Tim, just ask how it makes you feel. Because if it's something that you really are chomping at the bit to share with others on social media, then it may well be that why this study has become so popular is because of its appeal, not because of its accuracy.

What kind of thing might this be? Alex's first example is one we've looked at before on More or Less, the relationship between breastfeeding and the health of a baby. The idea that breast milk is better seems plausible, but is it true? So the World Health Organisation recommended exclusive breastfeeding for the first six months, based on evidence that this leads to better outcomes across a range of criteria –

So physical health outcomes for the child, mental development for the child, physical recovery for the mother, all of these were strongly correlated. But here's the thing, is that whether a child is breastfed is not random. It depends on family circumstances. So maybe the mothers who are able to breastfeed have a more supportive home environment because breastfeeding is tough.

And it might be that that home environment, not the breast milk itself, is driving the better outcomes.

And so when you control for home environment, other factors such as the mother's own IQ, whether the mother smoked, family affluence, you actually find that the link between breastfeeding and many of these outcomes goes away. So just to stick with topics that aren't going to annoy anybody, let's talk about inequality and the cost of inequality. You...

You talk in your book about the spirit level, which is, it feels like a blast from the past for me because we, I interviewed Kate Pickett, one of the authors of the spirit level back in, I think, 2009, maybe 2010, when that book was, was a huge bestseller.

And The Spirit Level is a book basically publishing a bunch of statistical evidence which links inequality, economic inequality, to social problems and health problems, crime, and so on. Now, I think a lot of people found that very plausible. You have your doubts, right?

Yeah, so again, this is an issue where the correlation is there in the data. So I'm not going to quibble with the correlation that they find it's not due to mismeasuring particular variables or doing something dodgy with the statistics. And the correlation is basically more unequal countries have more problems, basically. Yes. To dramatically oversimplify. There's not really any doubt that that does seem to be true.

Yes, and this is across a range of outcomes. And so this is why they interpret this as a strong evidence of inequality. So the outcomes may well be gross national happiness. It might be obesity and health. So this range of outcomes suggests there might be a unifying explanation, which is inequality.

But the issue here is again, what they've done is they found a correlation. They have put on an interpretation which many people might be willing to lap up because many people don't like inequality, myself included. However, there could be a whole host of other potential variables driving it and one alternative explanation could well be poverty.

Now, poverty is related to inequality, but it's not exactly the same thing. Because if poverty is the true cause, then the solution as a government is to bring up those at the bottom, not necessarily to clip the wings of the people at the top, unless that's a way of actually redistributing wealth to help the people at the bottom.

And so when you look at other people who've now controlled for poverty, they find that it's poverty which is driving the outcomes such as ill health and obesity, not necessarily inequality itself. Needless to say, that's not the view of the authors of The Spirit Level, but we're just going to leave that there and move on. MUSIC

For a final example, Alex has been looking at research into the relationship between how diverse a business is and how well that company performs. For example, you might look at the percentage of women or ethnic minorities on the board of directors or the executive team.

In this research, working out that first bit, the percentage of different people at the top of a company, is relatively straightforward. But the issue is that there's lots of different ways to measure financial performance. You could look at sales, you could look at profits, you could divide profits by the total assets, by the total earnings, you could take out accounting issues like depreciation, or you could look at total shareholder return.

McKinsey, the global management consultancy, has released a series of influential reports on this subject. Which showed apparently compelling

evidence that diversity is linked to performance. More diverse companies equals better performing companies. But there was a catch. The results were just not replicable when you looked at alternative measures of performance. So there was a replication which tried to look at six other equally plausible measures of performance and found that the results completely went away when you looked at the other five of the six.

And even in the one that they did report themselves, they found that the correlation was much weaker than was reported. But let's say, well, that was because they just didn't have the same data and they could have replicated it fully had they had the same data. The point remains that actually, if you look at all of the other plausible measures of performance, then you actually get nothing.

All of these are good things to think about. The complex statistical judgments sitting behind plausible public pronouncements. But it still leaves the question, what are people who aren't trained in statistical techniques meant to do about it all?

This is an important question because if the solution is you need to get a PhD in statistics, that's just unrealistic. People just don't have the time to do that. They don't have the time to look up every single reference and check every regression coefficient. But instead, what I'm trying to highlight in my book is that the skills to discernment are already within us. So whenever you see a study published on LinkedIn that people don't like the sound of,

there was no shortage of alternative explanations. Maybe it was correlation but not causation. Maybe this was just a hand-picked set of numbers. Maybe they looked at performance over one year and they should have looked at it over a larger time period. Maybe this is two smaller samples.

And so when you see a result that you want to be true, one simple trick I give is imagine you had the opposite result.

and see how you would try to explain it away. All we need to do is to unlock the natural discernment that we have anyway, which we already deploy selectively when we see things that we don't like, but the idea of imagining the opposite ensures that we also use that discernment even when we want something to be true.

Thanks to Alex Edmonds, author of May Contain Lies. And that's it for this week. If you've seen a stat in the news that you think needs a stern look, please do let us know. Our email is moreorless at bbc.co.uk. Until next week, goodbye.