In the classic book “Founders at Work” the founders of many high tech startups repeatedly return to the subject of failure and how they recovered from it. One of the popular conferences in San Francisco is FailCon.  Failure is regular subject of panel discussions at high tech conferences. That may sound strange to an external observer. Isn’t Silicon Valley all about innovation and success stories? Why such fascination with failure?

In truth, the popular perception of the Valley as a magical conveyor belt that churns out billion dollar companies from startups is just a trivial case of survivor bias. The Valley creates thousands of failed startups that go away without notice during same time it creates  a couple of Netscapes, Googles, and Facebooks. The latter get all the press and attention while the former disappear in the dark.  This creates an illusion of an unbroken string of successes. If one looks at the actual time spent by entrepreneurs, as a distinctively different class of people than salaried employees of successful startups, they spend the most of their time and effort creating, enduring, and recovering from failure rather then creating success.

The very first company I started failed with a great bang. The second one failed a little bit less, but still failed. The third one, you know, proper failed, but it was kind of okay. I recovered quickly. Number four almost didn’t fail. It still didn’t really feel great, but it did okay. Number five was PayPal.

Max Levchin (Cofounder, PayPal)

The real machinery of Silicon Valley creation resembles a complex chemical factory. Raw ideas and visions are funneled into it, and undergo multiple transitions from one stage to another in a high pressure startup environment. 90-95% break down and fail, and their raw materials recycle right back into furnace. The rest emerges from the silo years later as winners we all know today. One of Silicon Valley’s many secrets is the system of managing and dealing with systematic and constant failure of individual startups, while creating  almost guaranteed small percentage of winners in the end. Lets take a closer look at how this refinery of massive failures actually converts a few ideas into real successes.

Market Forces

One of the topics high tech pundits like to discuss is relative importance in startup destiny between initial idea, founding team, and the market. The answer usually puts lowest priority to an idea and splits the reminder of the  importance between the team and the market, often putting more emphasis on the team. Something like 55% team, 35% market and 10% original idea. Such analysis has some merits and is helpful for founders to maintain healthy state of mind while assailing huge odds stacked against them. However, after observing Valley life for 15 years this author realistic estimate would along the following figures: 70% market, 29% team, 1% original idea.

Lets start with simple example. Microsoft was floundering tiny company from 1975 till 1982. There is a rumor Microsoft was on verge of bankruptcy when 25,000$ check from then roaring Apple Computer for BASIC license saved the day. Things changed for the better when IBM deal revenues started to come in 1982 and the rest was history. Yet think on this timeframe. Its full 7 years. Google went from nothing to 23,000,000,000$ IPO in 6 years. It took Bill Gates himself to work like crazy for 7 years to just survive as tiny obscure startup.

What changed between ’75 and 82’? Did team get radically better? Did Bill Gates suddenly get much smarter? In reality all that changed was market. Instead of serving hobby market of selling BASIC and other niche software to early computer enthusiasts company changed its market – which was delivered to them on silver platter by IBM – to what would become most mass produced computing platform in history. Then the very same teams vent on to create unimaginable amount of wealth, which Google is yet to beat after 12 years of trying.

We can go on and on about this. Say Steve Jobs returning to Apple and reinventing the company. He does this by not trying to beat the markets Apple already lost to Wintel platform by late 90ties. Instead, he went to find new markets: high-end laptops, portable music players, smart phones where Apple unmatched design and elegance allowed them to dominate.

What this short excursion into history shows is that even the absolutely best of the best leaders of technology companies are:

• Powerless to change the outcome if they are in wrong market
• Can only succeed if underdeveloped market (smart phones) or new market (IBM PC) is created for them to enter.

Trick question: do you think founders of Silicon Valley startups are better or worse then Bill Gates or Steve Jobs to evaluate, pick and enter correct market when they just create first outline of their company? And what happens when they enter wrong markets? Simple – their company most likely will fail.

There is one little caveat to that whole marketplace discussion. We easily and confidently discuss metrics and timeframes of markets determined many years ago. The problem is we have some (not great) visibility into the past and virtually no visibility into the future.

Here is typical example from personal experience. Today we all know of multi-billion companies like Zynga formed on Facebook application platform. Yet how things looked when the platform was just opened up in summer of 2007? Back then we had no visibility whatsoever on platform future. Facebook was fairly small network, much smaller then dominant Myspace at that time. Nobody ever saw anything like app platform on social network, thus no industry record to compare it with. Recalling many conversations with I had with entrepreneurs and investors at that time opinions ranged from treating it as small curiosity not worth investing much time into (opinion of highly experienced VC who generated $800,000,000 returns from his investments) to guarded optimism it could be a good helper tool to bring audience to existing stand-alone websites.

The prediction that there will be its own vertical industry with dozen of companies, doing nothing but building facebook apps which will bring hundreds of millions in revenue and have market capitalization in billions of dollars was so out of the realm of conceivable reality nobody came even remotely close to predicting that. As we know now from many interviews of the Facebook insiders they also had no clue what was coming! Their thinking was it would be a nice side addition to the Facebook product to bring incremental growth for their user base. The platform explosive growth caught them completely by surprise, unprepared and lacking resources in many areas, thus creating very interesting experience for everybody who was on the Facebook platform the first year.

Lets look at this again. We have collective wisdom of Silicon Valley best and brightest estimating new high tech product (Facebook platform) and completely missing its importance and impact. The Facebook team and their venerated young founder completely missed the scale of their own product. So who the heck knows the answer?

The hard truth any Valley founder realizes sooner or later is that answer to such question is: nobody. Future is practically unpredictable and therefore markets are impossible to predict as well.

No amount of past experience or industry standing helps much to predict what is the best market or the best product to enter the next year. While bigger companies can deploy (barely useful) the tools like focus groups and market research initiatives, a typical startup won’t have even these dubious tools at their disposal. The only option remaining to them is to take a completely blind guess. 90%+ of time such blind guess will be wrong, and startup will either fail or – a very important option! – pivot to something new.

Market miss is probably the biggest and the most important factor in startup death rate. It is also practically unavoidable. Founders can be inhumanly smart or hard working, they can hire best team on the planet, and still fare no better then Bill Gates during ’75-82 or pre-iPod Steve Jobs.

Systematic failure is an irremovable part of startup ecosystem. Silicon Valley has naturally structured itself around constructive integration of such failure. Yet missing the market is not the only way in which startup can fail.

A Manual For Gold Seekers

A reader may get surprised at this point in our story. Did not we argue just recently how important is talent and how one must follow all these cardinal talent rules? Yet now author goes on to saying markets are everything, and team is almost an afterthought.

Not so simple. In that complex multi-layer refinery of startup lifecycle team does play a huge role. It just plays it later. The negative of being in wrong market are obvious – there are not that many customers, or not that much money, or there is no growth and momentum. Best team can blunt its pickaxe trying to make startup biz model work, and still most likely fail no matter how hard their try. However, what happens if they actually guessed correctly and find themselves in hot market?

Exploding new market has certain feel to it, it is like a digital Klondike. Nobody yet knows the future, but few prospects already have stricken rich gold veins (revenues, customers or growth) and there is a frenzy to grab the best parts of the market. That is where the team expertise and quality comes to the forefront. That is where “startup mode” of 24/7 non-stop marathon becomes typical, and the quality of people who can survive such inhuman wear and tear while keeping reliability to produce stellar results . I recall our own experiences during such supernova periods of startup life, and the most telling attribute that changes is one perception of time . Day of the week and time of the day largely loose meaning, and you start to think about the events in terms of “deliver +2 hrs from now”, “meetup +10hr from now”, “deploy +18hr from now”. The whole lifestyle becomes how to cram the absolutely most events in activities in the shortest timeframe and grabbing some sleep when you see a few hours not showered with the upcoming inner events. Sufficient to say very few people can survive such mode of operation for long, and one needs the whole team of them to run a successful startup.

There is one big caveat to all that. Remember Youtube sale to Google for 1,700,000,000$ ? Good outcome for #1 player in web video market. Quick, how much #2 fetched? Frankly, I have no idea. Probably you don’t either. The difference between #1 and #2 in such market, especially in consumer-facing industries is not just huge, it is astronomical. Sometimes #2 is good enough to get “just” 10-20 times less then #1 player. Most often #2, #3, …., #10 get all about the same amount – next to nothing. #1 takes the majority of the market and huge share of proceeds. Just ask Microsoft about their search revenue versus Google. Or their mobile share vs iPhone.

If you’re growing at 50 percent a year, and your competitor is growing at 100 percent a year, it takes only eight years before your competitor is 10 times bigger than you. And when it’s 10 times bigger than you, it can buy 10 times as much advertising and do 10 times as many projects and have meetings with 10 times as many customers. And you begin to disappear.

Joel Spolsky

What differentiates #1 from #2, #3,…#10 players is the team. What is ironic they did not even have to work twice or ten times better. 10% difference could be all it takes. The team works just slightly better and faster, customers flock to its product faster then they flock to competitors, that attracts more investors, more creative talent, success breads success, and before you know it such startup gets x 1,000 times better outcome.

Needless to say that fate of these #2, #3,…#10 players in the market is the same: absolute or relative failure. Guessing hot marketing is not enough. One got to build best team comparing to all others who guessed the same market. Yet even that is not the final story.

Pivotal Points

What about our tiny little 1% importance of the initial idea in startup? Why not make it 0% and say its completely irrelevant? The reason is that initial idea has significant yet indirect influence on both resulting market and team.

Imagine yourself in big dark forest in darkest hour of the night. Think of starting idea as very weak flashlight in very dark forest of big trees of unknowns: market, industry and overall future. You can sort of see where you going to make your next few steps next week. The future picture is quickly becoming blurry and  fuzzy, just some projections and schedules. Anything more then half year out is very dark, only few very big shapes (big vendor platform plans, broad trends) are visible in ray of that weak flashlight of initial planing. Starting idea is the direction where founders point that flashlight initially. Hence based on shapes and predictions that they recognize in the dark they make decisions about market they enter and team they recruit.

So all these predictions are unlikely to be true. So what do founders do when the first few steps with flashlight show an unpenetratable wall or  an unexpected object right on their projected path? They pivot and start moving flashlight slightly away trying to find the way around it. The initial idea starts to morph and change the shape. Paypal negotiated a big funding round as a payment system for the early Palm handhelds, and then a few days later pivoted into the web payments. Microsoft started with BASIC for Altair and then pivoted into operating systems.

By the time we had our first board meeting a month later, we had already realized that wasn’t going to to work…We started the board meeting basically saying “Hi, John. Hi, Pete” – the new VC guys – “We changed our business plan.” And these guys were like, “What?” They just put down $4 million to see something happen, and we said, “Sorry, we’re not going to do that; we going to do this.”

Max Levchin (Cofounder, PayPal)

The key understanding of pivot is that a startup cannot change its course completely. The existing team, existing products or prototypes if there is any, general expertise and knowledge of founders limit such freedom. That flashlight can only move to directions adjacent to the initial idea, and the speed of such transition is fairly slow. Even in extremely high energy startup environment people don’t like to give up something they worked months or even years on. It is harder to change existing habits even of a 6 months old startup than to create entirely new ones on a blank slate of a new startup.

Here we come to the main bottleneck of startup life, and therefore the main reason of such astonishingly high rate of startup failure. Look how all components are coming together. Founders no matter how experienced they are have very little chance of predicting and guessing future markets. All they have is tiny flashlight showing just where they putting their feet next few weeks or months. Initial idea realistically is just an excuse to form a team and find investment in general lightcone of that flashlight. The real most important event in startup life will happen when absolutely unpredictable opportunity in absolutely unpredictable market opens up and then it because death race for #1 stop against few other teams who also ended up in right place at right time.

Imagine an professional race runner who is training all his life to be fastest athlete on the planet. He is not just good, great or best. He is inhumanly good, his extra “supplements” are borderline to illegal substances to make his muscle work beyond any limit. There is no silver or bronze medal. It is #1 or nothing. Furthermore, he cannot just train on background and wait for Olympics. Instead, he must tire himself to death everyday working on his initial idea. And here is the clincher: At some random unpredictable moment at time, at some unpredictable market (with luck) adjacent to our Olympic runner initial idea will create an opportunity of all-or-nothing potentially huge win. The Olympic Games between all teams is officially on… yet nobody even gets the signal the games are started! Nobody will even signal clearly thats it is time to morph initial idea and pivot with all speed toward the Olympic Games. Spending extra 3-6 months working on originally designed product – “oh, we just got to ship it, and then we will see what is up with that new little windows/web/facebook ” – could be all the difference between #1 and #2 (if not #10).

Having an idea in a largely right general direction is not enough. Building and recruiting the best team in that market is not enough. Just being lucky of recognizing the right market is not enough. One has to have an incredible luck of having the perfect timing while having all other requirements lined up to succeed in order to finish line and claim #1 prize.

Successful startup is such an improbable event, that it is even astounding we have only 90% failure rate, instead of much more comprehendable 99.999%.

High Risk Ecology

The formation of the Valley culture took many decades. All that time market forces outlined above were fully active. Incidents built upon incidents to form the knowledge. Or perhaps more exact term would be meta-knowledge. Nobody could predict next market hit, yet VCs started to learn how many startups would fail even with the best proven founders as helm, although they could never know exactly which ones of them. Rockstar employees going from startup to startup saw how many best ideas backed by the best effort would fare and evolve. First pioneers passed some of these lessons to the next generations, which learned or re-learned them anew. Good habits sometimes helped startups to survive and these habits were more likely to end up in minds of founders and executives who made it big, and thus their lessons and speeches were eagerly learned by growing youngsters. Bad habits inescapably led to failed startups and had harder time to be replicated. It was Darwinian selection with meta-knowledge of running and operating startups evolving for more then 50 years and seeing many generations of investors and entrepreneurs. While good habits did not guarantee success at all, bad habits certainly led to failure. One way or another almost all of these bits of knowledge were connected with dealing with failure.

Lets take a broad look at ecology formed by living and breathing in 90%+ failure atmosphere:

• Failure is never ever penalized. On the opposite: the whole community of investors, advisers and related professions will constantly and strongly signal that any failure is just an attribute of experience and will do their outmost best to reinvigorate startup teams to try again.
• There is a mild penalization for wasting time. Spending extra 4-6 months on trying to revive a product or idea which clearly did not connect with the market even after a few iterations will be grumbled about yet tolerated. Delaying one year or more will be actively resisted and overridden. For first time startup team it is the hardest thing to admit that their vision actually failed. By forcing startups to fail fast , it gives them the chance to try something new and reapply they immense talent to something that may actually work.
• Operate at the maximum transparency. You don’t know the right market, the time when “Instant Olympics” start and how to adjust your product vision to it. The only cure is to open all the possible barriers for communications. Instead of building walls that isolate fragile startup from the outside, integrate it into all possible channels to the broader community. Conferences, industry pundits, bloggers, old friends from previous carrier, etc. Users, potential partners, investors and rockstar employees will be able to find you easily and quickly. Your seed ideas, which as we now know are most likely to be imperfect, will be chewed upon and criticized by other brilliant entrepreneurs thus giving you invaluable feedback. The most important benefit will be that you are much less likely to miss the signals about incredible new markets opening up, or about critical shortcomings about your current product or startup roadmap.

• Be flexible. Remember these old kung-fu movies? The main hero is kicked so hard that he flies across the room, yet somehow he just lands on his feet with no apparent damage and jumps back in the fight again. Startup life will be like that. By the very definition pretty much everything out there is bigger and more powerful then a startup. The market is unpredictable, the product’s future is unknown. The team make up depends on both factors, which most likely will change and multiple times. Investors may get tired of waiting. Your platform vendor may turn against you. Since you can not predict and be ready for any potential surprise the only option is to embrace flexible and agile workstyle. Concentrate on being as quick as you can in shorter timeframes (few weeks, couple month). As circumstances around you change, immediately adjust your plans and reiterate.
• Iterate fast. That just same advise about startups as whole reduced down in size for  internal operations. As we said failed startup is normal, it better to fail fast without wasting time. Why not to apply the same logic to your internal operations? Any project, experiment, research should have a clear criteria what it should achieve and what will be considered as a failure. If given milestone is not reached on time, have a clear team-level discussion if given data is valid indication of failure, and kill failing projects as fast as practically possible. Startup biggest sin is keeping the failed project around “oh, we spent so much time building it!” just for sentimental reasons while their upkeep distracts and defocuses the team from working on something really important.

• Bottom up instead of top down. The chaotic nature of describe process destroys any possibilities of operating old top down models. The unpredictability surrounding a new startup will break all layers of any top down model conceived in advance. Top down model are just in general not compatible to constant tweaking and iteration. Therefore a startup team after establishing broad area what kind of problems they trying to solve (flashlight cone) should concentrate and follow any seeds of early success (first customer, first users, first media excitement) and make up bigger plans as that success story unfolds. Even more then twenty years ago such thinking was becoming prevalent on much slower development cycles comparing to current lighting fast speeds of startup iterations.

International Lessons

America threw off the old world’s hostility to failed businessmen along with British rule. Back in the 1830s one of the things that most struck Alexis de Tocqueville about the country was “the strange indulgence which is shown to bankrupts”, which, he said, diverged “not only from the nations of Europe, but from all the commercial nations of our time”.

The Economist, Jan 7th 2010

Silicon Valley undoubtedly got it start due to the great leniency USA was traditionally showing to its failed businesses. After half a century of future development, that culture is amplified and focused many hundreds times more then it ever was before. The benefits of such approach impressive such as they are, are staggering when one considers compound interests of multi-generation selections of leaders and top talent such breding grounds reliably produces.

This lesson may be hard to learn for international community who is envious of Silicon Valley success. It is not just a method of operations that few enlightened executives can agree to adopt. It requires a broad acceptance by investors, government, employees, and to a degree by a larger public and especially media. European culture traditionally puts a high price on failure.

One common theme recurring in publications of international investors and goverment officials is that this or that country X desperately lacks entrepreneurs and startup-friendly talent. Lets take a look at that from perspective of young brilliant graduate of some international top institute. He is just beginning his professional carrier and has roughly two options. Join established company; say a bank or an oil company. He will get a great compensation, a full job security and a good carrier. However, he understands such choice will result in fairly boring and routine job. He is excited about the possibilities of doing a startup and researches the option further. Being brilliant, he quickly realizes that startup is a very chancy proposition. Even if in a great and remote Silicon Valley 9 out of 10 startups fail! Yet he is still willing to take even that chance. Then he finds out that the whole system is stacked up against him. Investors insist on grandiose top-down plans to validate the investment, which they actively meddle in and micromanage. Furthermore, they are very unfriendly to loosing money, with few founders even being afraid of criminal charges in case of failure. Society as whole is not tolerant to failure. While our young graduate will try and fail, his peers will be making good money in legacy industries and steadily climb corporate ladders. Estimating what is going to happen in the next few years our graduate realizes he runs a very real risk to be stigmatized by society, permanently harm his carrier and potentially even make some enemies just by trying a risky startup. Being brilliant he does the rational thing: he joins a big company.

In meantime, life in Silicon Valley goes on. In given cohort 10 startups start, and 9 fail. One winner captures everybody attention and international investors & governments are yet again amazed at the Silicon Valley creativity. Yet the main product created in that cohort goes unnoticed by the external world. These 9 failed startups with 5-10 person each, had just created 50-100 people tempered and experienced by the failure. Reinvigorated by Silicon Valley community these people will now jump back into fray, some to join other startups, some to start their own. The real full production of 10 startups was one winning startup and 100 high quality people. While representing the loss of individual investors (from which they are of course protected by portfolio system), it’s a big win for the Valley as whole. These people will bring new ideas, experiences and most importantly even higher tolerances to failure to any startup or company they join.

In Silicon Valley failure is constantly refined into materials of future success, and it is one of the most important components of the Silicon Valley system.