What is Regression to Mean?

A video explaining Regression to Mean by Rolf Dobelli the author of The Art of Thinking Clearly on the IDFC Mutual Fund Youtube Channel.

WHAT IS REGRESSION TO MEAN AND THE DUBIOUS EFFICACY OF DOCTORS, CONSULTANTS AND PSYCHOTHERAPISTS

His back pain was sometimes better, sometimes worse. There were days when he felt like he could move mountains, and those when he could barely move. When that was the case – fortunately it happened only rarely – his wife would drive him to the chiropractor. The next day he would feel much more mobile and would recommend the therapist to everyone.

Another man, younger and with a respectable golf handicap of 12, gushed in a similar fashion about his golf instructor. Whenever he played miserably, he booked an hour with the pro, and lo and behold, in the next game he fared much better.

A third man, an investment adviser at a major bank, invented a sort of ‘rain dance’, which he performed in the restroom every time his stocks had performed extremely badly. As absurd as it seemed, he felt compelled to do it: and things always improved afterward.

What links the three men is a fallacy: the regression-to-mean delusion.

Suppose your region is experiencing a record period of cold weather. In all probability, the temperature will rise in the next few days, back toward the monthly average. The same goes for extreme heat, drought or rain. Weather fluctuates around a mean. The same is true for chronic pain, golf handicaps, stock market performance, luck in love, subjective happiness and test scores. In short, the crippling back pain would most likely have improved without a chiropractor. The handicap would have returned to 12 without additional lessons. And the performance of the investment adviser would also have shifted back toward the market average – with or without the restroom dance.

Extreme performances are interspersed with less extreme ones. The most successful stock picks from the past three years are hardly going to be the most successful stocks in the coming three years. Knowing this, you can appreciate why some athletes would rather not make it on to the front pages of the newspapers: subconsciously they know that the next time they race, they probably won’t achieve the same top result – which has nothing to do with the media attention, but is to do with natural variations in performance.

Or, take the example of a division manager who wants to improve employee morale by sending the least motivated 3% of the workforce on a course. The result? The next time he looks at motivation levels, the same people will not make up the bottom few – there will be others. Was the course worth it? Hard to say, since the group’s motivation levels would probably have returned to their personal norms even without the training. The situation is similar with patients who are hospitalised for depression. They usually leave the clinic feeling a little better. It is quite possible, however, that the stay contributed absolutely nothing.

Another example: in Boston, the lowest-performing schools were entered into a complex support programme. The following year, the schools had moved up in the rankings, an improvement that the authorities attributed to the programme rather than to natural regression to mean.

Ignoring regression to mean can have destructive consequences, such as teachers (or managers) concluding that the stick is better than the carrot. For example, following a test the highest performing students are praised, and the lowest are castigated. In the next exam, other students will probably – purely coincidentally – achieve the highest and lowest scores. Thus, the teacher concludes that reproach helps and praise hinders. A fallacy that keeps on giving.

In conclusion: when you hear stories such as: ‘I was sick, went to the doctor, and got better a few days later’ or ‘the company had a bad year, so we got a consultant in and now the results are back to normal’, look out for our old friend, the regression-to-mean error.

Next:
Outcome Bias – NEVER JUDGE A DECISION BY ITS OUTCOME

Similar Biases:
Problem with Averages – WHY THERE IS NO SUCH THING AS AN AVERAGE WAR
Contrast Effect – LEAVE YOUR SUPERMODEL FRIENDS AT HOME
It’ll-Get-Worse-Before-It-Gets-Better Fallacy – WHY ‘NO PAIN, NO GAIN’ SHOULD SET ALARM BELLS RINGING
Coincidence – THE INEVITABILITY OF UNLIKELY EVENTS
Gambler’s Fallacy – WHY THE ‘BALANCING FORCE OF THE UNIVERSE’ IS BALONEY

Previous:
Incentive Super-Response Tendency – NEVER PAY YOUR LAWYER BY THE HOUR

The above article is from the book The Art of Thinking Clearly by Rolf Dobelli. The article is only for educational and informative purposes to explain and understand cognitive biases. It is a great book, definitely worth a read!