Early Retirement: The Median Won’t Have Enough

Early retirement isn’t easy to pull off in America unless you’re an outlier. Americans work forever: both because they have no other choice and because that’s what’s “normal”. I was talking to a family member recently that said he’ll work until he no longer can or drops dead. I get this attitude, especially if you like what you do. But not everyone does. If you’re in the position because you need to make money and not because you love what you do, it’s time to evaluate your options.

What defines early retirement? The Social Security Administration considers early retirement the age of 62, the age at which you can receive a reduced benefit. Medicare kicks in at 65 so some people consider early retirement prior to this age. Active duty military personnel can retire with half their pay after 20 years of service. Many civil servants retire in their 50’s because they get a pension and full health benefits after 30 years. For our definition in this article, we’ll consider early retirement 55 and under.

Why retire early? Early retirement can be a tactic in your overall strategy to live a better life. It doesn’t mean you’ll be sitting at the park all day eating butterscotch. Early retirement means achieving financial independence so that you can do what you want to do in life. Live life on your own terms. Take a bat to your alarm clock. Only wear workout pants and a ball cap.

Most people can’t just retire early. They have obligations: bills, kids in college, cars and on and on. You have to have a retirement plan in order to achieve an early exit. It takes someone looking at every aspect of your financial life in order to execute the plan. This includes Social Security planning, portfolio allocation, geo-arbitrage and home equity investing, passive versus earned income analysis, income structuring, etc.

Tax Savings in Early Retirement

You can save a ton in taxes if you retire early. Your taxes drop off a cliff when your income goes down. Check out the federal income tax rates by income bracket below.

Your tax rates won’t just go down on your federal taxes due either. Your capital gains and dividend income taxes will drop. Your payroll taxes will drop. If you live in the great state of Maryland like me, your state taxes will also drop (we have a tiered bracket like the federal government). And I’m not just talking about saving a few hundred bucks here; you could save tens of thousands in taxes due if you check out early.

Take a look at the below chart that shows marginal tax rates by income level for a married filing jointly couple. If you can manage to get your combined earned income below $76,000, your tax rate will be 15% and your tax rate on dividends and long term capital gains will be 0%. Now throw in some itemized deductions and exemptions and your tax paid will be negligible.

early retirement tax rates

Won’t my earned income go to zero when I retire? Not exactly. You’re still likely to do something, right? Maybe you want to teach voice lessons part time or consult for a tech company. You won’t stop helping people out in early retirement and so you’re likely to have some form of earned income. Depending on your spousal situation, you may also have Social Security income.

Let’s say your older spouse collects $30,000 in Social Security and you work part-time in early retirement and make $20,000. You also have passive income of $2,000 on municipal bonds. The amount of your household’s Social Security income subject to tax is only $2,500. That’s not $2,500 in tax. That means you will have have to pay tax on $2,500. Awesome! Social Security is a very tax-favored income source.

early retirement income


Beat the Median

This guy retired at 52 with a $3 million net worth and shared his experience. It’s hilarious. For example, he talks about how when he first retired, people would offer him jobs thinking that he was desperate to get back into the workforce. He had coffee with a friend of a friend and the guy spent the whole time figuring out how to get him a job. People just can’t accept that you don’t have to grind anymore.

If you have a $3 million net worth at 52, you’ve won. You can easily live off of your retirement income and you never have to work again. Why can’t people accept this? I think it has something to do with the fact that most people are never able to save enough to retire early. Very few people have saved enough for early retirement. Look at the below graph that shows the average retirement account savings by age group. The most recent data shows that the age group 50 – 55 have on average $125,831 saved in their retirement (Source: BLS).

early retirement amount saved

The “mean” savings or average isn’t always the best way to look at data. Average can be skewed by very high values. If you were put in a group of 10 people, one of which was Warren Buffett, the average net worth of the group might be $5 billion. The median net worth, on the other hand, might be something like $100,000.

Let’s take a look at median net worth by age according to the US Census Bureau. The blue bar shows net worth including home equity. The green bar shows net worth without home equity. The median net worth for someone in the 45 – 54 year old age bracket is $84,542 and drops to $25,006 if you exclude the equity they’ve built in their own home.

Median Net Worth By Age

It’s astounding to me how the median net worth drops off if you exclude home equity but it also makes sense. People are forced to save if they own a home. It takes a long time to build equity unless you were lucky enough to catch an appreciation wave. But by their early 50’s most people have a pretty sizable equity position in their homes. The problem is that home equity is not an investment; it’s more like a negative yielding savings account. A bit of home equity is a good thing. A lot of home equity is a poor way to allocate capital in almost all circumstances.

It doesn’t actually take much to be better than median. Most people reading this are above the median based on read demographics. Congrats to you! If you’re 50 and your net worth is $85,000, your slightly ahead of the median. However, to have a successful early retirement, you’ll need to have a net worth much higher than the median.

Early Retirement Success

I put out a spreadsheet that generates the net worth that you should target based on your age, location, and expected annual retirement expenses. Those who want to retire early should check it out to see if they’re on track from a net worth standpoint.

For example, if you’re 50 years old and live in Colorado, the spreadsheet says that you should have about $800,000 (based on $50,000 in annual expenses) as your net worth. The spreadsheet is based on the retirement goal date of 59 1/2.  If you want to figure out if you can retire at age 50, this is what you’ll need to do. You should input 59.5 as your age into the spreadsheet even though you’re 50. This yields a net worth target of $1.25 million. If your net worth is greater than or equal to $1.25 million, then you can retire!

What if you’re currently 48 and want to retire at 50? In this case, you just subtract 2 years from 59.5 and input that as your age in the spreadsheet. Setting your age at 57.5 generates a net worth goal of $1.15 million. That means that if you’re around that net worth, you’re on track to retire at 50. You see how this works now? Just take the years until you wish to retire and subtract from 59.5. That’s the number you’ll input as your age. Simple, right? A 30 year old who wishes to retire at 40 will input 49.5 as their age in the spreadsheet (59.5 – 10 years until retirement).

This spreadsheet generates a net worth goal you can use no matter your age. Just download a copy using the “download” link below and start playing around. If you don’t want to use the adjustment factor, type in “1” in the cell above the green net worth target cell.

Do you plan to retire early? If so, at what age and how far are you away from that age? Are you on track? If not, what are you doing to get there?

By | 2017-04-24T07:05:59+00:00 April 24th, 2017|Retirement Planning|2 Comments


  1. Corey April 25, 2017 at 1:38 pm - Reply

    Loving the posts. Keep em coming.

    • Matt Miller April 25, 2017 at 6:24 pm - Reply

      Thanks Corey! I’ll have at least one per week for as long as people keep coming.

Leave A Comment