Should I Pay Off My Mortgage?

If you have a question like this one, send it in. I’ll tackle case studies that have educational value.

“I am anxious to pay off my mortgage since, with the current standard deduction, there is no advantage in claiming mortgage interest.

“My wife and I are retired. I am 72, with a pension along with Social Security, and have $850,000 in my IRA. I have a mortgage balance of $134,000. To get that after tax I would have to take a distribution of $185,000, which obviously will reduce my portfolio dramatically.

“Is this a good move? My return on investment with Fidelity has been 10-15 percent annually with a 60/40 mix of stock and bond funds.”

Michael, Missouri

Readers send in queries like this. I’m going to be answering the ones that illustrate tricky tax and investment decisions.

My answer to the Missourian:

Good move? Probably. Retirees should pay off their mortgages. You’re lucky to be in a position to do that.

For many people, no doubt you included, taking out a mortgage in order to get into a house turned out to be a good decision. But we have to deconstruct home ownership. A mortgaged house is two things, an asset and a liability. Having a house is a good investment. Having a mortgage is a bad investment. The goal of a retiree should be to have a house without a mortgage.

The 40% of your IRA in bond funds means you are a lender. If the funds track the U.S. bond market then a good portion of your savings is being lent out, at low rates, to the U.S. Treasury. This part of your portfolio is earning 2% at best. Your mortgage is probably costing you 3% or more.

Borrowing at 3% in order to lend at 2% is a bad idea.

Two things cause people like you to hesitate before cashing in an IRA in order to pay down a debt: the taxes they’d owe and the IRA returns they’d miss.

Yes, the IRA withdrawal means writing out a check to tax collectors. You’re probably in a 27.4% bracket (state and federal combined), so you’re going to owe $51,000 on a $185,000 withdrawal.

But taxes on this money are inevitable. If you are past 59-1/2 (the cut-off to avoid penalties) and not expecting to see your tax bracket go down, postponing the inevitable does not leave you better off. If the IRA grows, so do the tax bills.

The arithmetic becomes clearer if you rethink what an IRA is. Where you see an $850,000 asset, I see something different. I see you as the custodian for an account that has two beneficiaries. You’re sitting on $617,000 that belongs to you and also on $233,000 that already belongs to tax collectors.

Look at what growth does to this account. If, for example, you’re able to double the portfolio at Fidelity, the account will then have in it $1.7 million. Of this, $1,234,000 will belong to you and $466,000 will belong to the tax guys. You’ve doubled your money and you’ve doubled the government’s money.

In effect, what you have is not an $850,000 asset but a $617,000 asset that’s all yours and that grows tax-free.

What, then, are you sacrificing when you take a big distribution? Assuming you take it out of the bond portion of your portfolio, you’re losing a return that comes to 2% pretax and, thanks to the wonders of IRAs, the same 2% after taxes.

And what are you gaining by ripping up the mortgage? You’re getting a guaranteed return of 3% before taxes. Thanks to the wonders of the standard deduction, you’re not deducting interest and that 3% mortgage is costing you the same 3% after taxes. So getting rid of a mortgage earns you 3%.

There it is. Paying off the mortgage costs you an aftertax 2% and earns you an aftertax 3%. It’s a winning move. It would still be a winner, albeit a more modest one, if tax rules change and you go back to deducting interest.

Now let’s tackle the other reason people stick with 3% mortgages, which is that they are investing money to earn 10% or 15%. This is a faulty comparison. High returns come from risky assets like stocks. The mortgage is a sure-thing liability (you can’t duck the debt), so it must be compared to a sure-thing asset (a loan to the U.S. Treasury).

The apples-to-apples comparison comes into sharper focus when I hypothesize that your entire $185,000 withdrawal comes out of low-risk bonds. At this first stage of your financial makeover, then, the stock funds aren’t touched.

Now you take a look at what’s left and see a Fidelity account that has a high percentage in stocks. Is that allocation too high? Maybe, maybe not. But that’s a separate discussion.

Selling bonds to pay off a mortgage leaves you better off no matter what happens to the stock market. Meanwhile, whether you have too much money in the stock market is an independent decision that shouldn’t influence your thinking about the mortgage.

Unlike comparing 2% to 3%, determining the correct level of risk for a 72-year-old is not a question that has a clear answer. Taking money out of stocks would lower your expected return but might be wise anyway. What are your living costs and how well are they covered by pensions and Social Security? Would your retirement survive a stock market crash with the portfolio you have now? Have a talk with your wealth advisor about this.

Whatever you do, don’t compare 10% stock market returns to 3% mortgages.

I said, above, that the mortgage paydown is probably a good move. Now here are some things to be cautious about.

First, your tax bracket. You may need to carve up the $185,000 distribution into thirds, spreading it over 2022-2024, in order to avoid being kicked from a 22% federal rate into 24%.

Next, your near-term plans. Any chance you’ll be moving to Texas or Florida? If so, hold off on excess distributions until you’re out of reach of the 5.4% Missouri tax.

Last, your end game. Is there a good chance that a diminished IRA will run dry while you’re still healthy enough to live independently? Would you at that point be averse to moving out—to a rental or to a smaller house—in order to extract some cash? And would you, in order to stay put, probably use a reverse mortgage to cover monthly expenses? If this outcome is likely, and if your existing mortgage has a lot of years to run, you should perhaps hang onto it. Its terms are much better than anything you’d get on a reverse mortgage down the road.

Do you have a financial situation like this? Send a description to the address listed in my bio. Include a first name and a state of residence. Include enough detail to generate a useful analysis. Letters will be edited for clarity and brevity; only some will be selected; the answers are intended to be educational and not a replacement for professional advice.