What is Intention-To-Treat Error? | Understanding Cognitive Biases

WHAT IS INTENTION-TO-TREAT ERROR AND HOW SPEED DEMONS MAKE SAFE DRIVERS

You’ll find it hard to believe, but speed demons drive more safely than so-called ‘careful’ drivers. Why? Well, consider this: the distance from Miami to West Palm Beach is around 75 miles. Drivers who cover the distance in an hour or less we’ll categorise as ‘reckless drivers’ because they’re travelling at an average of 75 mph or more. All others we put into the group of careful drivers. Which group experiences fewer accidents? Without a doubt, it is the ‘reckless drivers’. They all completed the journey in less than an hour, so could not have been involved in any accidents. This automatically puts all drivers who end up in accidents in the slower drivers’ category. This example illustrates a treacherous fallacy, the so-called intention-to-treat error. Unfortunately, there is no catchier term for it.

This might sound to you like the survivorship bias (chapter 1), but it’s different. In the survivorship bias you only see the survivors, not the failed projects or cars involved in accidents. In the intention-to-treat error, the failed projects or cars with accidents show up prominently, just in the wrong category.

A banker showed me an interesting study recently. Its conclusion: companies with debt on their balance sheets are significantly more profitable than firms with no debt (equity only). The banker vehemently insisted that every company should borrow at will, and, of course, his bank is the best place to do it. I examined the study more closely. How could that be? Indeed, from 1,000 randomly selected firms, those with large loans displayed higher returns not only on their equity but also on their total capital. They were in every respect more successful than the independently financed firms. Then the penny dropped: unprofitable companies don’t get corporate loans. Thus, they form part of the ‘equity-only’ group. The firms that make up this set have bigger cash cushions, stay afloat longer and, no matter how sickly they are, remain part of the study. On the other side, firms that have borrowed a lot go bankrupt more quickly. Once they cannot pay back the interest, the bank takes over, and the companies are sold off – thus disappearing from the sample. The ones that remain in the ‘debt group’ are relatively healthy, regardless of how much debt they have amassed on their balance sheets.

If you’re thinking, ‘OK, got it’, watch out. The intention-to-treat error is not easy to recognise. A fictional example from medicine: a pharmaceutical company has developed a new drug to fight heart disease. A study ‘proves’ that it significantly reduces patients’ mortality rates. The data speaks for itself: among patients who have taken the drug regularly, the five-year mortality rate is 15%. For those who have swallowed placebo pills, it is about the same, indicating that the pill doesn’t work. However – and this is crucial – the mortality rate of patients who have taken the drug at irregular intervals is 30% – twice as high! A big difference between regular and irregular intake. So, the pill is a complete success. Or is it?

Here’s the snag: the pill is probably not the decisive factor; rather, it is the patients’ behaviour. Perhaps patients discontinued the pill following severe side effects and so landed in the ‘irregular intake’ category. Maybe they were so ill that there was no way to continue it on a regular basis. Either way, only relatively healthy patients remain in the ‘regular’ group, which makes the drug look a lot more effective than it really is. The really sick patients who, for this very reason, couldn’t take the drug on a regular basis, ended up populating the ‘irregular intake’ group.

In reputable studies, medical researchers evaluate the data of all patients whom they originally intend to treat (hence the title); it doesn’t matter if they take part in the trial or drop out. Unfortunately, many studies flout this rule. Whether this is intentional or accidental remains to be seen. Therefore, be on your guard: always check whether test subjects – drivers who end up in accidents, bankrupt companies, critically ill patients – have, for whatever reason, vanished from the sample. If so, you should file the study where it belongs: in the trash can.

Next:
News Illusion – WHY YOU SHOULDN’T READ THE NEWS

Previous:
Fallacy of the Single Cause – THE STONE-AGE HUNT FOR SCAPEGOATS

The above article is from the book The Art of Thinking Clearly by Rolf Dobelli. The article is only for educational and informative purposes to explain and understand cognitive biases. It is a great book, definitely worth a read!