Secrets of a superforecaster

By Andrew Rumbelow, Segment Head: Institutional Business

Michael Mauboussin, Wall Street investment strategist and head of Global Financial Strategies at Credit Suisse in New York, recently engaged us at a special client luncheon in Cape Town. In sharing his insights on investment skill, he drew on the findings of psychology researcher, Philip Tetlock and spoke about a topic that has captured the minds of thousands of analysts in the last year:  a rare breed of ‘superforecasters’, people who consistently make superior predictions.

Beware the over-confident forecaster
Tetlock embarked on a 21-year study of expert political forecasters, and then kept track of their predictions. He studied the aggregate accuracy of 300 experts making thousands of forecasts of specific political and economic outcomes over this time, looking for patterns in their comparative success rates. On average, these forecasters performed little better than chance. But when confronted with evidence of their futility, they ‘put up their psychological defence shields’ and made rationalizations about their flawed predictions. Instead of saying, “I was wrong,” they would insist, “I almost got it right…”.

 “How you think matters more than what you think.”

Two statistically significant findings emerged from the study:

• the more ‘visible’ and the more media mentions the pundits had, the worse their predictions were (ie, there is an inverse relation between fame and accuracy)
• the quality of the predictions had less to do with age, gender and political persuasion and more to do with their thinking.

Says, Mauboussin, it’s a matter of judgement style, and alluded to Isaaih Berlin’s analogy of hedgehogs and foxes. He characterised judgment styles into hedgehogs (who have one big idea, exude great confidence and review everything through this single world view) and foxes (who know a little bit about a lot of different things). Hedgehogs don’t care when they’re wrong and they tell very compelling stories; they are typically famous but poor predictors. But foxes tend to be better forecasters, they learn from their mistakes and are sceptical about grand theories, modest in their forecasts, and ready to adjust their ideas based on actual events. These Tetlock termed ‘optimistic sceptics’.

No surprises, the aggregate success rate of foxes is significantly greater… especially in short-term forecasts. And hedgehogs typically fare worse than foxes, especially in long-term forecasts. They are blinded by their extensive expertise and lofty theories. Foxes win not only in the accuracy of their predictions but also the accuracy of the likelihood they assign to their predictions.

Bottom line, says Mauboussin… The expert who bores you with his ‘buts’ and ‘exceptions’ is probably right about what’s going to happen. The charismatic expert who exudes excess confidence and has a great story to tell is probably wrong.

The secret to forecasting
In another study between Tetlock and the US Intelligence community, they identified a group (in the top 2%) of “superforecasters,” people who consistently make superior predictions that were vastly beyond what chance would dictate. How did they do it?

Tetlock discovered two common threads among them.  The first is that foresight is a real, measurable skill that can be cultivated through high persistency, and an ability to not rely too heavily on intuition, which can lead to poor decisions. The second is how these forecasters think, how they gather information and adjust their beliefs. They are all actively open-minded, intellectually humble, numerate, thoughtful updaters, self-critical, comfortable with a sense of self-doubt but curious and hard working. For investment organisations, however, it is more than just these personality traits. It is also how they work in teams (teams can be better than individuals, but only under the right conditions), it is their awareness of their behavioural biases (such as overconfidence and loss aversion) as well as unlearning these, and finally, developing algorithms to improve successful outcomes (“some people are better at it. Just overweight their forecasts…”).

Importantly, superforecasters are not intellectual geniuses. They simply have a better ability to think logically and rationally (what Mauboussin terms RQ – high rationality quotient). They are actively open to considering views other than their own, and they embrace feedback.

Regarding behavioural biases, Mauboussin argues that we often fall victim to simplified mental traps that prevent us from coping with complex realities inherent in important judgement calls. These cognitive and behavioural errors are, however, preventable (he elaborates on this in his book “Think Twice”).

Good forecasters weigh the probabilities
Investors need to train themselves to consider a sufficiently wide range of outcomes by paying attention to the leading indicators of ‘inevitable surprises’. Always be aware you could be wrong.  Says Mauboussin, decisions are a matter of weighing probabilities and to get the best possible payoff, we need a careful calibration between confidence, conviction and probability. An appreciation of this ‘uncertainty’ is very important for money management. When allocating capital, portfolio managers need to consider that unexpected events can and do occur. Expressing things in a probabilistic way also allows some psychological cover against behavioural quirks such as loss aversion, where we love to be right.

One must balance the probability of an outcome (likelihood) with the outcomes payoff (magnitude of return). Probabilities alone are not enough when payoffs are skewed. This is compounded by loss aversion, as we like to be right as mentioned earlier in this article, and hence often seek high-probability events. A focus on probability is sound when outcomes are symmetrical, but completely inappropriate when payoffs are skewed. Even when information is incomplete, we must still make decisions based on an intelligent appraisal of available information. Here good process is especially important.

Process over outcome
Mauboussin says that while results are ultimately what count, we should never underestimate the value of process and its ongoing scrutiny. The best long-term performers in any probabilistic field, such as investing, all emphasize process over outcome. Because of probability, good decisions will sometimes lead to bad outcomes and bad decisions will sometimes leads to good outcomes but over the long haul, process dominates outcome. He elaborates on this in his book “More than you know’.

The skill-luck continuum
In his book “The Success Equation”, Mauboussin explains that successful investing necessitates both skill and luck, with the reality being that a lot of investing successes fall more toward the luck side of the continuum. Determining whether an investment manager has skill involves understanding the effects of outliers (people who achieve extraordinary success, distinctly separate from the norm), reversion to the mean (the theory that a value will revert to its historic average over time), and the paradox of skill (as skill increases, luck becomes more important). Ultimately, says Mauboussin, for investment managers to demonstrate true investment skill, they need a high degree of persistent and predictable performance, and a healthy combination of both skill and luck. These skilful investment managers must also have a solid process, which needs to include analytical, behavioural and organizational components.

In the world of investing, we have seen consistently that whenever there is an outlier in terms of performance, there is almost always a combination of high skill and high luck.  Paradoxically, in activities that involve some luck, improving skill makes luck ever more important in determining those outcomes. On an absolute basis, skill has never been higher in today’s world while luck has become increasingly more important.

Last but not least, the best investment leaders recognize that proper, even bold, action requires sound thinking and judgement. To improve the quality of your forecasts, keep brutally honest score. Enjoy being wrong, admit to it and learn as much as possible from it. And always communicate with clear probabilities attached. “Forecast, measure and revise: it is the surest path to seeing better.”

Sanlam Investments Institutional Business would like to thank Michael Mauboussin of Credit Suisse for his contribution to this article.