‘I feel like I’m investing just to afford taxes and broker fees.’ I’m 72, have a cushy amount of money invested and was withdrawing 4% a year, but once I pay fees and California taxes that won’t be enough. Should I ditch my financial adviser?

Question: I have a comfortable balance in my investment account but have a question about only taking out 4% of my total each year (if necessary). Totaling my federal and state taxes (I live in California), and paying estimates for the next year, plus adding in my broker fees actually totals 4% of my investment balance.  Basically I feel like I’m investing to pay taxes and broker fees!  I’m 72, house is paid for, and I’d like to travel, but jeez. What should I do?

Answer: There are a few things to consider, but first let’s look at the fees you may be paying to the broker.  It’s pretty standard to pay about 1% of your assets under management to the financial adviser, explains certified financial planner and certified public accountant Maggie Klokkenga of Make a Money Mindshift. That said, often these fees are negotiable (read our guide on how to negotiate your financial advisers fees here) and many brokers have lowered their fees in recent years, says Alana Benson, investing spokesperson at NerdWallet. In short, you likely could pay less — and you can use this tool to get matched with a financial adviser who might meet your needs here.

Have a question about investing or your financial adviser? Email [email protected]. 

Unsure what you pay in fees? “Check your account statements to see what your account fees are. On the statement, if you divide the account fees by your total fair market value of the balance listed, you’ll have a general idea of the fees charged by the firm. If you don’t see any fees, check the next month,” says Klokkenga. This is because investment advisory firms often charge their fees on a quarterly basis, and there may be other fees, such as transaction fees, each time a buy or sell is done.  “If you’re paying trade fees or high expense ratios, it may be worth looking into other brokers with lower fees,” says Benson. Your broker should also be able to provide transparency with his or her structure and all fees that you pay. 

If the manager’s performance is underperforming, you should look to change advisers to ensure the management they are providing is worth the fee, pros say. “If this account is managed, you should evaluate the performance of the manager over their tenure of oversite on the portfolio, based on the appropriate portfolio benchmark net fees,” says Phil Picariello, CFO at Zingeroo, a social investment platform. (Looking for a new financial adviser? You can use this tool to get matched with a financial adviser who might meet your needs here.)

Another option? You could ditch the adviser entirely. “You could also look at lower cost investment vehicles if they are appropriate based on investment goal, time horizon and risk tolerance. Depending on your investment experience, you could move some or all of your assets to a self-directed brokerage account or IRA and manage the assets yourself,” says Picariello, who adds that this does take “time and diligence.”

A deeper look at the 4% rule

The 4% rule states that a person hoping to be retired for the next 30 years could spend 4% of their portfolio in the first year of retirement, with inflation-adjusted withdrawals added to that percentage each following year.  It’s just a rough guideline, and it has come under fire recently. What’s more, as Vanguard points out: “the 4% rule didn’t factor investment fees into estimated returns” and “minimizing costs allows for a significantly higher likelihood of success.” We agree on that point, which is why we lead this story with a review of your fees, and how to get them down. Now, moving forward, let’s talk taxes, which is another major factor eating into your money.

Taxes eating into your nest egg

The tax situation is another issue you’ll want to tackle. First up, California is a high-tax state. “High state taxes can make it enticing for retirees to explore other areas and living in a state with lower taxes means more of your money stays in your accounts,” says Benson.

It sounds as if you may have your money in a traditional IRA or 401(k) or something similar, and the fact of the matter is that you do have to pay income tax in the year you withdraw the money. That reduces what you have left to spend. And since you’re 72 now, you had to start taking the required minimum distributions — which are simply a minimum amount you must withdraw from your account each year (or risk high penalties). Those withdrawals “will be included in your taxable income except for any part that was taxed before (your basis) or that can be received tax-free (such as qualified distributions from designated Roth accounts),” the IRS explains. 

If this is all making your head spin, you’re not alone.  And some financial firms — as part of their fee — will help you navigate the tax situation, explains Ryan Townsley, certified financial planner at Town Capital: “If I can save a client six figures in taxes over the course of a retirement, this pays for me many times over,” says Townsley. But, of course, you need to evaluate whether your firm is helping with that, and whether that is worth the fee you pay. If not, it may be time to find a new adviser or do it on your own.  

Have a question about investing or your financial adviser? Email [email protected]. 

Questions have been edited for brevity and clarity.