4
Taxes Down the Drain
In January 1994, I spoke to a small gathering of the superrich at the behest of the Northern Trust Bank of Chicago. I was a speaker at their Family Financial Forum, which was held at an extremely opulent hotel in Dallas.
“Family Financial Forum” sounds like a nice little gathering, perhaps with tea and cakes, that you might have one evening at your local church or community center. But that’s not what this Family Financial Forum was. The average wealth of the families represented at this forum was $400 million. This was not your typical community group.
The wealthy were represented either by family members themselves or by their staff, or both. Most superrich have family foundations managed by hired staff. A few of the people at the conference bore names that would be recognized for their wealth, but most would not be known to the general public.
Northern Trust spared no expense rolling out the red carpet for these, their most well-to-do clients. There were two “name” speakers. One was Dick Cheney. At that time he was known primarily for his understated televised briefings as secretary of defense during the Gulf War in Iraq. He was frequently mentioned as a possible presidential candidate. He seemed, in person as in his television briefings on the Gulf War, understated and modest.
The other name speaker—who may have cost Northern Trust even more as a speaker than Dick Cheney—was political commentator Norman Ornstein. Ornstein was in great demand for his biting political wit and insight.52
Getting to the Meat
Those two speakers were the entertainment; the rest were the meat. The meatiest—and probably the longest speaking and most attentively heard—was a man from a big law firm who spoke on estate taxes. In his talk he displayed, and explained as best he could, the most complicated chart I’ve ever seen at a conference—save for conferences intended only for technical people. The chart was about how to create a tax avoidance strategy for a wealthy extended family. As I recall, a lot of it involved generation-skipping trusts. Apparently, if you can arrange to give a lot of money to your grandchildren but not to your children, you can avoid a lot of taxes.
It’s not surprising that there was so much interest in avoiding taxes. Members of the U.S. private sector do not have friendly feelings toward taxes. They try to avoid them whenever they can. And this particular group included many people who were the least friendly toward taxes of all.
My Talk Bombs
Given this fact, it may seem odd that my talk might have been the least well-received one at the forum. I don’t think it was a bad talk. I think, actually, it was rather good, if a little dull—but no more dull than estate taxes. I thought I made my small number of points clearly and well.
But the main thrust of the talk seemed to fly in the face of what people in attendance believed or wanted to believe. My talk was on “After-Tax Investment Returns.” It doesn’t sound very controversial, but it seemed to make some people indignant.
The main point of my talk was simple: “What you get is what you keep.” That is, you have to take all costs into account before you decide whether an investment strategy is a good one or not. In this particular case, I was focusing on the tax cost. Some investment strategies cost you more in taxes than others. It pays to consider the comparative tax cost when you compare two investment strategies.
I gave a good example of two real-world mutual funds. One of them, Fund A—the Franklin Growth fund—had an annual return over the period 1963–1992 of 9.6 percent. Fund B—the Seligman Common Stock fund—had a much better return, 11.2 percent. Consequently, if you had invested $1,000 in each one of them, before taxes you’d have $24,400 with Fund B but only $15,800 with Fund A.53
You’d think Fund B was a better investment than Fund A. But after taxes the story was much different. Fund B had high turnover—that is, stocks were frequently bought and sold. This meant the owner—if taxable—would have to pay short-term capital gains taxes at a high rate on the gains every year. Fund A, by contrast, had low turnover. The owner of Fund A had less in capital gains and would have to pay taxes only at the much lower long-term capital gains rate.
Because of this the owner of Fund A would have $10,800 after taxes, while the owner of Fund B would have only $8,400, even though—before taxes—the owner of Fund B had $4,600 more. That extra $4,600—plus $2,400 more—got eaten up by taxes.
I also pointed out that according to a study over the same time period by two Stanford professors, the average that taxable mutual fund investors lost to taxes was almost 3 percent annually. But you could reduce that greatly if you used—or if your investment manager used—a tax avoidance strategy.
To my surprise the audience reception felt icy. I thought my delivery was poor. Maybe my tie had a big stain on it from lunch. I wasn’t sure what was wrong. One woman in the front row seemed particularly fidgety. Finally, she blurted out, “Well, isn’t that the tail wagging the dog?”
She explained that paying attention to taxes would divert the manager’s attention away from the most important thing, which was getting the best possible before-tax return.
I Keep My Mouth Shut
Could I have replied to her with the actual truth, which is that the manager’s efforts to get the best possible return were probably a waste of time? Could I have pointed out that study after study shows that investment managers—even the ones who cater to the super-rich—don’t beat a portfolio composed by a blind Ouija board operator? No, that would render her speechless with horror and would bring a complaint to a highly attentive Northern Trust executive vice president, perhaps to the president. That would be my last talk to the superrich at an extremely opulent hotel.54
She was not the only person in the room who felt the way she did. To put the best spin on it, it was obvious from the general reaction that my message was ahead of its time. (In fact, that turned out to be the case. A perception of the importance of after-tax investing began gradually to dawn on some investors—though still not most—in subsequent years.)
This is my interpretation. Like most people in the room, the woman had a warm professional relationship with her investment manager, a person of good social standing and a fine manner. She extended feelings of trust and respect to that person for his supposed investing skills and his ability to keep confidences. She may possibly have gotten wind of rumors that professional investment management, on average, adds no benefit, but she could not believe that applied to her investment manager. She may have somehow detected in my talk the slightest suggestion that behind my emphasis on after-tax returns, there was a deemphasis of what may have been, in my view, the discredited quest for superior before-tax returns. She couldn’t, and wouldn’t, believe that.
The belief among many of the wealthy that they can have special access to superior money management becomes almost a part of their identity. Even to hint at questioning that premise is to eat away at their egos. Nothing else explains the stiff reaction I got.
The Dereliction of Duty of Money Managers
If you do not agree with the woman’s premise—that an investment manager must not think about tax avoidance because it might interfere with the execution of his investment strategy—then you must conclude that the failure of investment managers to avoid taxes is nothing short of a dereliction of duty.
Tax avoidance doesn’t matter, of course, if the investments are in a tax-deferred vehicle like a pension fund, endowment fund, 401(k), or IRA account. But if the investments are in a taxable account, then tax avoidance should be the first and foremost consideration of an investment manager. The ability of an investment manager to enhance a client’s bottom line in this manner exceeds anything else he can do. (Fees are at least an equal consideration, of course, but that is the concern of the client and not the manager, whose concern is arguably to maximize his revenue without losing the client.)55
But the vast majority of investment managers do not give taxes a thought. Evidence of this fact lies in the extraordinarily high turnover of most mutual funds.
The average turnover in equity mutual funds is now nearly 100 percent, according to the mutual fund–monitoring service, Morning-star, Inc. (it was much lower a few decades ago). That means most of the stocks in the fund are sold during a year, and others bought in their place.
A Free Loan from the Government
If a stock is held for less than a year and it gained during the year—as most stocks do in most years—then the gain is taxed at the income tax rate. For investors of above-average income, that rate is about 30 percent or more. Hence, if a stock gained 10 percent during a year, 3 percent out of that 10 percent goes to the government.
Therefore, a manager should have an awfully good reason to sell a stock before it is held for a year, because 30 percent of the gain will be lost if he does.
If the stock is held longer than a year and then sold, the gain will be taxed only at the long-term gains rate. The long-term rate is 15 percent for most investors—half as much as the short-term gains tax rate.
But there’s even more reason not to sell a stock and realize a gain. The tax you pay on selling a stock is a tax you have to pay now. But if you keep the stock and sell it later, you can invest the tax money in the meantime.
The difference is huge. Deferring taxes has been called “getting a free loan from the government.” If you can pay a tax later instead of now, it’s like getting a zero interest loan. That’s a very valuable thing to have.
Suppose, for example, you defer a $1,000 tax for twenty years that you would otherwise have to pay now. If you just invest it in certificates of deposit (CDs) at 4 percent, you’ll have $2,191 in twenty years. Then after you pay the $1,000 tax, you’ll still have $1,191, whereas you would have had nothing if you’d had to pay the tax now.
Furthermore, if you defer the taxes long enough, you could realize a really important benefit—you could die! Then the taxes won’t have to be paid at all. This is not a joke. It can be a valuable thing for your heirs to inherit your investments before they are depleted by gains taxes.56
The vast majority of mutual fund managers, not to mention hedge fund managers, pay absolutely no attention to these obvious facts. As a result, they lose for their clients, on average, about 1 to 2.5 percent of their money every year to the government, when with a little effort they could reduce that to as little as 0.5 percent.
That may not be an awful thing in a global sense. If wealthy investors don’t pay extra taxes on their investments, someone else will have to make up the difference. But it should be an awful thing to admit to if you’re the client’s financial servant, charged with getting as much money into the client’s pocket as possible. You’re not doing that if you’re letting 1 or 2 percent of it get away unnecessarily every year. Do you plead that you’re making up for that loss with superior before-tax returns? Not a chance! That’s a claim with not a scintilla of evidence to back it up.
Once again the Big Investment Lie is used to cover a multitude of sins—the lie that you can beat the market by a whole lot if you’re smart and well paid. As soon as that lie is discovered to be illusory—like the emperor’s imaginary clothes—the embarrassments of high taxes and high fees are revealed in all their nakedness.
Tax Avoidance Strategy in Investing
Capital gains taxes make up a very substantial part of United States tax revenues. One of the reasons the U.S. budget was hundreds of billions of dollars in surplus in the late 1990s, then hundreds of billions of dollars in deficit in the early 2000s, was because of capital gains taxes. When the stock market was soaring in the late 1990s, racking up increases in the 20 to 30 percent range, capital gains tax revenues poured in. Then when the market dropped in 2000, 2001, and 2002, these revenues dried up.
Obviously, somebody has to pay U.S. federal taxes to provide funds for the federal budget. But in the U.S. system—and all other systems, too—no individual, and no corporation, wants to pay taxes, and everybody tries not to. It’s not considered shirking to try to minimize your taxes. It’s considered good fiscal responsibility and good business sense.57
Hence, advisors who help you minimize taxes are thought of as upstanding citizens and are well respected. The estate tax attorney who gave the complicated talk at the Family Financial Forum in Dallas was certainly a well-respected citizen and keenly sought after for his expertise.
Investment managers and advisors are also well respected and keenly sought after. The conference attendees’ investment managers could, in some cases, have saved their clients as much in taxes as the estate tax attorney, if they had put their minds to it. But that was not thought to be their job. Neither they nor their clients thought it was their job. Their job was to beat the stock market. Never mind that they could save their clients far more in taxes than they could ever make for them trying to beat the stock market. The Big Investment Lie—that you can beat the market by a lot if you apply expertise and effort—means that expertise and effort are not applied where they really count.
The biggest problem, as usual, is that the best investment strategy is also the simplest. It is so simple that you wouldn’t appear to need expert and hardworking managers and advisors to do it. Consequently, people who like to have expert, hardworking, and high-paid servants at their beck and call, and managers and advisors who like to have big inflows of revenue, don’t consider that strategy.
The best tax avoidance policy in investment is a simple, well-diversified buy-and-hold strategy. If you hold stocks for a very long time, you won’t have to pay tax on the gains, at least not for a very long time. And if you also want broad diversification, a passive index fund does the job quite nicely. Or you can try to use the strategy associated with investor names like Warren Buffett: choosing a portfolio of good companies and holding them forever. But if you aren’t sure you can choose as well as Buffett (and who can be sure—not even Buffett himself), it’s better to spread your investments across the whole market.
There are also more complicated tax avoidance strategies in investing. You can deliberately realize losses—that is, sell stocks at a loss—in order to offset gains. But these strategies have limited applicability. If you keep doing them for very long, you’ll wind up with a portfolio concentrated in only a small number of stocks that have never declined. It’s not a good investment strategy for the long run.
The best long-run strategy for tax avoidance is the simplest: buy and hold. You can do that by investing in a collection of stocks that you choose or in a passively managed, diversified, low-cost mutual fund. It doesn’t just save you a little in taxes. It can save you in cash an amount equal to 2 percent of your portfolio every year. If your portfolio is a million or more, that will save you upward of $20,000 a year immediately and much more than that over time. If your portfolio is $100,000, it still saves you $2,000 every year. That’s not peanuts.58
That’s a big tail, wagging a very small dog. The small—let’s put it plainly, nonexistent—dog is the gain an investment manager can add by constantly buying and selling stocks like most mutual funds do, somehow winding up not even beating a passive market index.
The Bottom Line: Tallying Up the Costs
As I’ve already shown, many investors use high-cost investment services. Most of them don’t even realize they’re using high-cost services. These services typically cost 2 to 3 percent of an investor’s assets annually.
To make things worse, higher-fee services tend to cost investors more in commissions than low-fee services and often in taxes as well, because higher-fee investment vehicles tend to have higher turnover.1
A Direct Comparison
Let’s compare two investors, John and Mary. Each has $250,000 to invest. Neither one will touch the money for thirty years until after they are retired.
John considers that he knows little about investing. So he thinks he will be “smart” and hire a good advisor—or a “wealth manager,” as the advisory firm he chooses calls it.
Mary decides to just invest 80 percent of the money in total market index funds purchased from Vanguard or Fidelity or one of a small number of other low-cost index fund providers. Mary decides to put 70 percent of this 80 percent in a domestic U.S. stock fund and 30 percent of it in an international fund. The other 20 percent of her money she will invest in a Vanguard intermediate bond fund.
John’s comfort level with stocks is about the same as Mary’s. So the wealth manager will advise him also to make a similar allocation of his portfolio to stocks and bonds—though the wealth manager will in all probability recommend different funds or investment vehicles, and a larger number of them.59
Before fees and taxes, John and Mary will get about the same return over the thirty years—other than a small and unpredictable difference, plus or minus. Let’s assume that return will average 8 percent annually.
Thus, their results before fees and taxes will be essentially the same. After fees and taxes, however, the difference between John’s and Mary’s results will be very large—astonishingly so.
Mary’s total fees will be about fifteen “basis points”—or 0.15 percent per year. Her taxes will be a little over 0.5 percent a year.
But John’s total fees will be more. They will range somewhere between 1.2 and 2.8 percent greater than Mary’s. John’s taxes will also be greater than Mary’s. They will range between 0.1 and 1.7 percent greater than Mary’s.2
The results will amaze you. I will summarize them in the following table at the bottom of the page.
The table speaks for itself. After fees, Mary will have somewhere between 40 and 121 percent more than John—with a midrange of 79 percent more—depending on how costly John’s “wealth manager” and the investments he recommends are.
After both fees and taxes (if John and Mary are not able to put their money in tax-deferred accounts), Mary will have between 45 and 263 percent more than John—with a midrange of 137 percent more—by the time they need to access the funds. Mary will have at that time somewhere between $636,000 and $1,496,000 more than John. Of this amount, between $796,000 and $1,424,000 of John’s money will have gone to pay his wealth manager, and between $292,000 and $523,000 will have gone to the federal government in taxes, while only $103,000 of Mary’s money will have gone to pay fees and $349,000 to pay taxes.360
What John’s and Mary’s $250,000 will be worth in 30 years with…
Now who do you think was smarter, John or Mary?
As silly as it sounds, the fees paid to the investment advice and management industry are largely based on the following circular logic: “We pay them a lot, so they become wealthy; they are wealthy, so we think they are expert and knowledgeable; we think they are expert and knowledgeable, so we pay them a lot.”