Understanding Survivor Bias and How It Can Kill Your Business

The owners who closed their schools don’t speak at conventions. That absence is the most expensive thing in your business, and it has a name.

Every convention, every summit, every mastermind has a stage. On that stage are the owners who made it. You can shake their hands, buy them dinner, ask them anything you want.

The owners who closed their businesses aren’t there. They didn’t buy a ticket. Nobody scheduled a breakout session called “What I Got Wrong.” They sold the mats, took the sign down, went to work somewhere else, and every lesson they paid for went with them.

That absence has a name. It’s called survivor bias. Once you can see it you’ll find it in your curriculum, your pricing, your marketing, your retention numbers, and in most of the advice you’ve ever been given about running a school.

I wrote in July about the two pieces of advice that keep school owners broke. This article is the mechanism underneath that one. Error A and Error B are both a form of survivor bias. Learn the idea itself and you’ll build your business instead of getting stuck on ideas that may not work.

Everything below is aimed at a school. If you want the idea on its own terms, with the research, the investing and the everyday versions, I wrote a fuller explanation of survivor bias that covers the ground this article only touches.

Before you take one more piece of advice, ask how many people tried the same thing and what happened to all of them.

What is survivor bias?

The best illustration comes from World War II. Abraham Wald (Mangel & Samaniego, 1984) worked with the Statistical Research Group at Columbia on a question the Army Air Forces cared a great deal about: where do you put armor on a bomber?

Armor is heavy. You can’t armor everything. The sensible thing is to look at the planes coming back and map the bullet holes. Wings. Fuselage. Not much on the engines.

The obvious answer is to armor where the holes are. Wald’s contribution was noticing that the sample of planes they looked at was made entirely of aircraft that survived. Damage in those places was survivable. That isn’t a theory, it’s demonstrated, because the plane you’re measuring is sitting in front of you. The places with no holes were the places where a hit meant the plane never came back to be measured.

(The popular version of this story is tidier than the history. Wald wrote a series of technical memoranda on estimating vulnerability from selectively observed survivors. The internet version compresses it into one dramatic moment in a briefing room. The mathematics is real.)

The missing planes carried information. So do the closed schools.

Why does this fool intelligent people?

Three things stack up.

Memorable examples are easy to retrieve, so they feel common. Once you know how a story ended, the path to that ending looks obvious and planned. And we turn messy, partly random histories into clean stories with a hero and a turning point, because a story is far easier to teach than a pile of numbers.

None of that requires anybody to lie. An owner who built a great school can sincerely believe one principle caused it while underweighting the timing, the corner they leased, a competitor who closed, a head instructor who happened to be extraordinary, and plain luck. Their explanation is honest testimony about their experience. Whether it explains the cause is a separate question, and nobody asked it.

What does the math actually look like?

Here’s the smallest example that shows it. Take 1,000 owners. One hundred succeed, nine hundred don’t.

Now the claim: 80 of the 100 successful owners used strategy X. That’s 80%. It’s true, it’s verifiable, and it’ll get a standing ovation.

Run it the other direction. Owners who used X: 880 of them, and 80 succeeded. That’s 9.1%. Owners who didn’t use X: 120 of them, and 20 succeeded. That’s 16.7%. There’s more evidence that NOT using strategy X is the better choice!

Strategy X is the most common behavior among the winners and it’s associated with roughly half the success rate. Nothing in the original claim was false. The denominator was simply missing, and the denominator was the entire answer.

If you only pay attention to success, you’re giving yourself misinformation.

“Every successful owner works 70-hour weeks.” Does that hold up?

This is the one that costs owners their health, so it’s worth running the numbers on. Same 1,000 owners. Now count hours instead of strategies.

Look at the top row on its own and the case is closed. Eighty-five of the hundred successful owners are grinding 70-hour weeks. Any speaker can put that on a slide.

Now the second row. Ninety percent of the owners who failed were working those same hours. Run the rates: among the 70-plus crowd, 85 of 895 made it, which is 9.5%. Among the ones working under 70, 15 of 105 made it, which is 14.3%.

The hours were what everybody was doing, winners and losers alike, and a behavior common to both groups is the one thing that cannot have split them apart. Something else did the work, and grinding out another ten hours a week won’t find it for you.

This advice does real damage, because it points an owner at the one thing they fully control. They add hours, the number doesn’t move, and they conclude they need more hours. The person who gave them that advice believed every word of it, and was quoting a number with no denominator attached.

Has this fooled people who should have known better?

In Search of Excellence came out in 1982. Peters and Waterman selected 43 excellent American companies and derived the attributes they shared (Peters & Waterman, 1982). Within about two years BusinessWeek ran a cover story titled “Oops! Who’s Excellent Now?” Atari, one of the 43, had essentially collapsed.

Twenty years later Tom Peters wrote a piece in Fast Company called “Tom Peters’s True Confessions.” (Peters, 2001) In it he says the list came from asking colleagues at McKinsey and other smart people which companies were doing cool work, and then working backward to quantitative measures. His words: “Okay, I confess: We faked the data.” He has since said that line was sharper than he meant it, and that varying research measures isn’t the same as fabricating numbers. Fair enough. The selection was still a room full of people naming companies they admired.

Good to Great came out in 2001. Jim Collins selected 11 companies whose stock returns had already been exceptional and catalogued what they had in common: humble leadership, discipline, focus (Collins, 2001). Circuit City filed for bankruptcy in 2009. Fannie Mae went into federal conservatorship in 2008.

The problem sits in step one of both, not in the later failures. The sample was defined by the outcome. Companies with identical humility and discipline that went under were never eligible for either study, because going under removed them from the pool before the research began. Both books could establish what great performers had in common. Neither could establish that those traits separated great performers from anybody else.

Phil Rosenzweig named the error in The Halo Effect (Rosenzweig, 2007): the delusion of connecting the winning dots. He also answers the natural defense, the one that says these authors gathered enormous amounts of data. If the sample is selected on the outcome, the volume of data doesn’t repair it. You can interview a thousand survivors and still learn nothing about what killed the others.

What other famous advice was built the same way?

Built to Last (1994). Collins and Porras surveyed hundreds of CEOs to identify 18 visionary companies, then wrote up the habits those companies shared (Collins & Porras, 1994). Within about a decade a large share of the list had slipped badly, Motorola, Ford, Sony, Disney, Boeing, Nordstrom and Merck among them. The companies were picked because they had already won.

The Millionaire Next Door (1996). Stanley and Danko interviewed hundreds of millionaires and reported the habits they found: live below your means, drive a used car, invest steadily (Stanley & Danko, 1996). Sound advice, and I follow most of it. But nobody interviewed the far larger group who lived below their means, drove a used car, invested steadily, and never got there. Nassim Taleb went further in Fooled by Randomness (Taleb, 2001), pointing out that the sample also caught people who happened to invest during one of the strongest bull markets in history. Two filters, not one.

The 10,000-hour rule (2008). Gladwell popularized it from research on elite performers, and the finding is real as far as it goes: people at the top of demanding fields have put in enormous practice. What that cannot tell you is how many people put in 10,000 hours and never reached the top, because the study started with the ones who did (Gladwell, 2008).

When Macnamara and colleagues ran a meta-analysis across many fields (Macnamara et al., 2014), deliberate practice accounted for about 26% of the variation in games, 21% in music, 18% in sports, 4% in education, and under 1% in professions.

That last number deserves a second read. In professional work, hours of practice explained less than one percent of the difference between people. Practice still matters, and you should still do it. It is nowhere near the whole story, and the reason we believed it was the whole story is that we only ever studied the winners.

Notice what these four books have in common. Enormous research budgets, careful authors, real data, and the same broken first step. If it can happen to McKinsey partners and Stanford professors with a research team, it can happen to you at a Sunday seminar with a legal pad.

“We have twelve success stories.” What’s wrong with that?

Same twelve stories. Same real people. Same verifiable results. Watch what the enrollment number does to them.

The advertisement is identical in all three rows. At 30% you should be paying full attention. At 0.6% the program is collecting people rather than producing successes, and the twelve who did well would most likely have done well anywhere.

Ask for the enrolled count. It’s the first number that matters and the last number anybody volunteers. If a program won’t tell you how many people started, you’ve already learned the thing you needed to know.

What about a ten-year track record?

Line up 1,024 advisors and assume every one of them is guessing. No skill at all. Pure coin flips.

Year one, half of them happen to be right. Year two, half of those. Keep going.

At year ten one advisor is left holding a flawless ten-year record built out of nothing at all.

He isn’t lying. The record is real and you can check every year of it. It’s also worthless as evidence of skill, because a room full of pure guessers was guaranteed to produce roughly one of him. The mistake isn’t believing the record. The mistake is never asking how many people started the contest.

This is exactly why the finance industry stopped tolerating survivor-biased fund databases. Funds that perform badly get merged or shut down and disappear from the record. Study only the funds still listed and the entire category looks better than it ever was, so serious analysis now requires databases that keep the dead ones.

Our industry doesn’t keep the dead ones.

Where does survivor bias show up in a school?

Your curriculum. Ask your black belts what kept them training and you’ll get thoughtful, useful answers. You’ll also get answers from the only people who didn’t quit. The students who left at yellow belt are holding the information you actually need, and they’re not in the building. Their answers are in your cancellation records, assuming you kept any.

Your advisors. Covered in July. One great school is a sample of one, selected on the outcome, with the location, the market, the staff, the timing and the person all varying together. Three schools in three towns with three different teams varies exactly the things that need varying. That is a crude experiment rather than a credential, and a crude experiment beats a great story every time.

Your marketing. If your team deletes losing campaigns and keeps a folder of winners, you’ve destroyed your own denominator. A winning ad only means something measured against the ones that didn’t work. Keep the losers. They’re not clutter, they’re your control group.

Your retention numbers. Surveying current students tells you why current students are happy. It can’t tell you why anybody left, because everybody who left is gone. Take every student who enrolled in a given month and follow all of them, including the ones who disappeared. That one change will tell you more than any survey you run this year.

Your ceiling. That one needs its own section.

What about the owner who has been in business twenty years?

I said in July that years in business qualify nobody. Here’s the reason.

An owner who has been flat for ten years is a survivor. Survival makes them visible, accessible and credible, because tenure reads as expertise and it’s easy to mistake one for the other. Their conclusion usually arrives as a ceiling claim: this is what the business pays, this is what the market supports, this is as good as it gets.

That claim is extremely well supported inside their sample. Their sample is themselves. They’re in Phase C, where the owner makes peace with the plateau and calls it a lifestyle business.

There’s a second effect here that’s less obvious and worth the ninety seconds it takes to understand, because it’ll change how you read your peers.

A school can stay open for ten years two different ways. Strong systems, or easy conditions: cheap rent, no real competitor, a protected corner, a spouse’s income covering the gap without comment. Either one is enough to survive.

Look at the group of owners who are still open after ten years and you’ll find excellent operators grinding it out in hard markets alongside mediocre operators sitting comfortably on great corners. Set those two side by side and it looks like systems don’t matter much. They matter enormously. You’re just looking at a group that was assembled by survival, and survival accepted either ticket.

Should I just ignore successful people?

Once people learn about survivor bias they sometimes swing to dismissing every success story. That’s worse than where they started.

A survivor shows you that a strategy is feasible. They’ll show you operational detail you’d never have invented, combinations you had not considered, and ideas worth testing. What they can’t give you is the odds. A testimonial tells you something is possible. Only a denominator tells you how likely it is.

Compare instead of collecting. Use the winners to generate ideas, then go find the people who did the same thing and didn’t make it. If the comparable failures didn’t use the practice, you have something worth acting on. If they used it just as often, you’ve found a habit of successful people rather than a cause of success.

I said it in July about the single-school owner and it holds here. Keep them close, and keep them out of your business plan. Beat them, and have fun doing it.

What do I actually ask?

1. How many times have you produced this result, in how many places, with different people running it? Once, here, with them on the floor, is a story.
2. How many people have you taught this to, and what happened to all of them? Not the successes. All of them.
3. Did you do this before the result, or notice it afterward? Practices discovered in hindsight get promoted to causes without ever earning it.
4. What has to be true about my situation for this to transfer? “It works for everybody” means it works no matter who you are, where you are, or who is teaching your classes. Almost nothing in business is like that.
5. What would I see if this advice were worthless? Ask it out loud and watch what happens.

The fifth question is the strongest and almost nobody asks it. An explanation that can’t be wrong is a weak one. It’ll absorb whatever result you get, it’ll never be revised, and you’ll spend years assuming the failure was yours.

Before you take the next piece of advice

Find the graveyard. In your school it’s the cancellation records, the students who tested once and never came back, the campaigns that never ran twice, the programs you stopped mentioning. Most owners delete all of it. Keep it. It’s the only comparison group you’ll ever have that costs nothing.

Then run the five questions on the next person who tells you how to run your business. Including me.

Find a mentor who has built it, rebuilt it, and built it again somewhere new. Then ask them what didn’t work.

The full research behind this article, including the statistics, the psychology, the causal logic and a survivorship audit you can run on your own business, is in the complete report. It is a free PDF you can download and keep.

And if you want to go deeper than the school floor, the full explanation of survivor bias runs the same idea through science, investing, social media and everyday risk. It is the broader version of everything here.