Why Comparing Your Portfolio to the S&P 500 Is a Losing Game

11/21/2023

6 minutes read

2023.11.21 07:22

As I wrote this blog, the is up roughly 17% year-to-date. Most likely, your portfolio isn’t. This is a common frustration among many investors in the market this year in particular. As discussed previously, the S&P 500 index performance is a bit deceiving. The majority of the gain in the market this year has come from essentially seven stocks with the largest concentration in the index in terms of market capitalization.

S&P 500 Contribution Point

The surge in those stocks has skewed the performance of the broad market index. The performance of the bottom 493 stocks remains markedly different.

Mag-7 Market Cap

As shown, the market capitalization of the top seven stocks is so large that it skews the performance of the index overall. We can see this visually by comparing the performance of the market and equal-weighted S&P 500 indices.

Market Cap Vs Equal Weight Index

This market bifurcation may not change in 2024 if Goldman Sachs is correct in their estimates.

“Consensus expects the Magnificent 7 will continue to deliver faster growth than the rest of the index. Analyst estimates show the mega-cap tech companies growing sales at a CAGR of 11% through 2025 compared with just 3% for the rest of the S&P 500. The net margins of the Magnificent 7 are twice the margins of the rest of the index, and consensus expects this gap will persist through 2025.

From a valuation perspective, the Magnificent 7 trade at a large P/E premium vs. the rest of the market, but relative valuations stand in line with recent averages after accounting for expected growth. The Magnificent 7 trades at a P/E of 29x, 1.7x the 17x P/E multiple of the median S&P 500 stock. This ratio ranks in the 91st percentile since 2012. However, on an earnings-weighted basis, the Magnificent 7 long-term expected EPS growth is 8 pp faster than the median S&P 500 stock (+17% vs. +9%).”

Mag-7 Sales Growth vs Rest of S&P 500

Why am I telling you this? Well, when the end of the year comes, and you look at your performance relative to the S&P 500 index, you will likely be disappointed.

However, that is precisely what Wall Street wants you to do.

Wall Street Wants You To Compare

Comparison is the root cause of more unhappiness than anything else. Perhaps it is inevitable that human beings, as social animals, have an urge to compare themselves with one another. Maybe it is because we are all terminally insecure in some cosmic sense.

Let me give you an example I discussed with Adam Taggart at Thoughtful Money last week.

Assume your boss gave you a new Mercedes as a yearly bonus. You would be thrilled until you learned everyone in the office got two. Now, you are upset because you got less than everyone else on a “relative” basis. However, are you deprived on an absolute basis of getting a Mercedes?

Comparison-created unhappiness and insecurity are pervasive. Social media is full of images of people showing off their lavish lifestyles, giving you something to compare to. No wonder social media users are terminally unhappy.

The flaw of human nature is that whatever we have is enough until we see someone else who has more.

Comparison in financial markets can lead to awful decisions. For example, investors have trouble being patient and letting whatever process they have work for them.

For example, you should be pleased if you made 12% on your investments but only needed 6%. However, you feel disappointed when you find out everyone else made 14%. But why? Does it make any difference?

Here is an ugly truth. Comparison-related unhappiness is for Wall Street’s benefit.

The financial services industry is predicated on upsetting people so that they will move money around in a frenzy. Money in motion creates fees and commissions. The creation of more and more benchmarks, products, and style boxes is nothing more than the creation of more things to COMPARE with. The end result is investors remain in a perpetual state of outrage.

The lesson we want to drive home here is the danger of following Wall Street’s advice of beating some arbitrary index from one year to the next. What most investors are taught to do is to measure portfolio performance over a twelve-month period. However, that is absolutely the worst thing you can do. It is the same as being on a diet and weighing yourself every day.

If you could see the whole future before you, making an investment decision knowing your eventual outcome would be effortless. However, we don’t have that luxury. Instead, Wall Street suggests that if your fund manager lags in one year, you should move your money elsewhere. This forces you to chase performance, creating fees and commissions for Wall Street.

We chase performance because we all suffer from the 7th deadly sin – Greed.

Most of us want all of the rewards without regard for the consequences. However, instead, we should learn to “love what is enough.“

In a year like 2023, where primarily seven companies drove the S&P 500 index, many individuals, thinking they “missed out,” will want to change their strategy for next year.

As is often the case, such will likely be a mistake.

Goldman Sachs May Be Disappointed

The table below from Callan Investments is an excellent example of the risk investors take by chasing last year’s best-performing sector. If you pick any asset class, you will see they are rarely the top performer for long. A good example in 2023 was that “cash” won, and the S&P 500 index was down 18%. If you had chased last year’s best-performing asset class, you would have woefully underperformed the S&P 500 index in 2023.

Investment Returns Table

While Goldman expects the S&P 500 index to have another winning year, as we noted in analysts are often wrong, and by a large degree.

“This is why we call it ‘Millennial Earnings Season.’ Wall Street continuously lowers estimates as the reporting period approaches so ‘everyone gets a trophy.’”

The chart below shows the changes in Q4 earnings estimates from February 2022, when analysts provided their first estimates.

Analysts Estimates for Q4-2023

But while Goldman is very optimistic about earnings growth in 2024, the rest of the analysts community has already started cutting their previous estimates for next year.

2024-Estimates Over Time

Given still elevated interest rates, tighter lending standards, and slowing wage growth, there is more than a substantial risk of slower economic growth next year. While such would lower inflation, it will also reduce earnings growth, suggesting 2024 could be a lower return year for the S&P 500 index.

Conclusion

When you sit down at the end of the year to analyze your performance, I suggest not looking at just 2023 as your benchmark. Investing aims to achieve a rate of return over a long period to reach your financial goals. Therefore, look at the average rate of return you have achieved over the last 5-years and compare that to your goal. This will give you a better sense of how you are doing and reduce the potential for emotional mistakes.

For example, over the last 5-years, the S&P 500 equal-weighted index has returned on a nominal basis 43.21% versus 57.05% for the market-cap weighted equivalent. However, during that period, the returns from the equal-weighted index came with lower volatility, allowing you to stay invested during more troubling times. More importantly, if you need a 6% rate of return to reach your retirement goal, even though the equal-weighted index underperformed in 2023, the 8.6% average return still has you ahead of your objectives.

S&P 500 Market Cap vs Equal Weighted Returns

Financial Resource Corporation summed it up best;

“For those who are not satisfied with simply beating the average over any given period, consider this: if an investor can consistently achieve slightly better than average returns each year over a 10-15 year period, then cumulatively over the full period they are likely to do better than roughly 80% or more of their peers. They may never have discovered a fund that ranked #1 over a subsequent one or three-year period. That ‘failure,’ however, is more than offset by their having avoided options that dramatically underperformed.

For those that are looking to find a new method of discerning the top ten funds for the next year, this study will prove frustrating. There are no magic short-cut solutions, and we urge our readers to abandon the illusive and ultimately counterproductive search for them.

For those who are willing to restrain their short-term passions, embrace the virtue of being only slightly better than average, and wait for the benefits of this approach to compound into something much better.”

If you want to be a better investor, do what most investors don’t: