Fighting investors' greatest enemy: overconfidence

"I always thought the brain was the most wonderful organ in my body; and then one day it occurred to me, 'Wait a minute, who's telling me that?'" — Emo Philips

Financial analytics firm Dalbar calculates the actual returns earned by investors compared with the S&P 500. Its latest reading is cringeworthy: The average stock investor earned an average annual return of 3.83% from 1990 to 2010, vs. 9.14% for the S&P 500.

That gap is more than can be explained by management fees or the underperformance delivered by the average mutual fund's poor skills. Something else is eroding investment returns.

And that something else is you.

You buy when you shouldn't. You sell when you shouldn't. You think you're capable of doing things you probably aren't. You are, in other words, overconfident in your skills as an investor.

You've probably heard of the Lake Woebegon effect with drivers — the vast majority of drivers claim they have above-average driving skills. This even holds true for drivers surveyed in the hospital after being injured in car accidents that they caused.

The same overconfidence affects investors.

Markus Glaser of Munich School of Management and Martin Weber of the University of Manheim once asked a group of investors a simple question: How have you done at investing?

Just like drivers, more than half assumed they outperformed the average investor.

But the study found something even more disturbing. The researchers asked investors to estimate their annual returns, and then compared those estimates to the investors' actual returns by checking their brokerage statements. Investors were quite literally clueless about how their investments performed, overestimating their returns by more than 11 percentage points per year. The average investor painfully lags an index fund and thinks he's Warren Buffett, basically.

Part of this is understandable. Investing is hard. We spend untold hours talking with advisors, researching new ideas, watching CNBC, and listening to pundits. Most investors have uncomfortably little to show for their effort, so they resort to convincing themselves otherwise. Jason Zweig writes in his book Your Money and Your Brain:

By fibbing ourselves, we can give a needed boost to our self-esteem. After all, none of us is perfect, and daily life brings us into constant collision with our own incompetence and inadequacies. If we did not ignore most of that negative feedback — and counteract it by creating what psychologists call "positive illusions" — our self-esteem would go through the floor.

We're also overconfident because hindsight bias fools us into thinking big events like the financial crisis were easy to predict, and thus will be easy to predict in the future. In his book Thinking, Fast and Slow, Daniel Kahneman writes:

Our tendency to construct and believe coherent narratives of the past makes it difficult for us to accept the limits of our forecasting ability. The illusion that we understand the past fosters overconfidence in our ability to predict the future.

When you are overconfident, all sorts of dangerous behaviours arise that throw your investing results off track.

For one, your predictions will likely become less accurate. Philip Tetlock, a psychologist at U.C. Berkeley, studied expert predictions and found that those who were most confident in their forecasts actually had the worst track records. Confident forecasters tend to have broad, unwavering views about how the world works — think of investors who were assured hyperinflation was right around the corner — while those who make good predictions know that the world is more nuanced and are constantly updating their views.

As confidence rises, your perception of risk also diminishes. The best example of this is the hedge fund Long Term Capital Management, a team of investors stacked with PhDs and Nobel laureates who became so confident in their ability to predict markets that they borrowed US$250 for every US$1 of their investors' money … and went broke soon after. Financial advisor Carl Richards says, "risk is what's left over when you think you've thought of everything else." When you're overconfident, you're not thinking about much to begin with.

Overcoming overconfidence is easier said than done. But two things might help.

One, become fiercely objective when measuring your success as an investor. Don't just assume you've done well because the Dow is at an all-time high and your portfolio has gone up, too. Measure exactly how you've done. Most investors will be surprised to see their returns versus a benchmark — some pleasantly, others humbled off the ledge of overconfidence.

Two, talk to someone about your investments who is in a different emotional state than you are. The odds of making a good decision while in the heat of panic, euphoria, or any other emotion tied to overconfidence are low. I run most of my financial decisions by fellow Fool Matt Koppenheffer. He thinks differently than I do and is more level-headed in areas I tend to get emotional about. It's a tremendous benefit, and I encourage everyone to find a "finance buddy" with whom to do the same.

Every investor needs confidence. But Mae West wasn't right: Too much of a good thing isn't always wonderful.