Why Do Some Companies Thrive While Others Fail?

What makes some businesses sprout, grow, adapt, and succeed, while most never get off the ground? Researchers in the growing field of organizational ecology say it is not enough to study the companies that thrive. Answers lie in the stories of failure.

Bitten by the technology bug and drawn by the scent of riches, 10 men got together in a garage on Alma Street, Palo Alto, to launch a start-up company. They never made it. The venture collapsed before any sustained manufacture of their dream product.

The year was 1908, and the new, new thing that drew these entrepreneurs was the internal combustion engine. Their firm, the Stanford Automobile and Manufacturing Co., shared the same fate as the Divine Motor Car Co. of Chicago, McHardy of Detroit, and more than 1,600 others that entered the industry between 1900 and 1920, only to fade into oblivion.

For one group of scholars, however, the very existence and varied fates of these myriad enterprises do matter. With the dogged determination of census takers, these researchers enumerate whole populations of organizations—not just the Fords and Chevrolets of an industry, but also the lesser-known and short-lived businesses that make up the majority of firms.

They call their approach “organizational ecology.” This perspective tries to capture the full range and diversity of corporations, through their birth, growth, transformation, and mortality. Organizational ecology yields insights into how industries develop and change over time. Many of the findings in the field have challenged conventional wisdom about competition, demanding the attention of policy makers and business leaders alike.

In December, 2002, proponents of organizational ecology will celebrate the discipline’s 25th anniversary at a conference to be held at Stanford GSB. The Farm has provided a particularly fertile ground for scholars working within this research tradition. Organizational ecology can trace its birth to a 1977 paper coauthored by Stanford sociologist Michael Hannan, who is now the StrataCom Professor of Management at the Business School.

One of Hannan’s students, Glenn Carroll, has been prolific in adding to the organizational ecology literature. Carroll, who is the School’s Lane Professor of Organizations, in turn taught William Barnett, now a colleague in strategic management and organizational behavior—creating, so far, a three-generation chain of organizational ecology scholars under one roof at the Business School’s Knight Building. “They’re calling me the grandfather of the conference. They joke that it’s my retirement event,” says Hannan, hastily adding that he is not quitting anytime soon.

The research program grew out of a mounting unease with organizational studies’ prevailing focus on large, dominant firms. Hannan’s intuition was that there is a lot of diversity within industries, with hundreds of firms that people don’t notice. He also took issue with the assumption that organizations are plastic and changeable—if so, why is failure so common? Organizations are characterized by inertia, he felt, and there are good reasons for this. Reliability and accountability are valued attributes of organizations. These qualities are strengthened by predictable routines and structures, which create inertia as a byproduct.

Hannan was looking around for conceptual models for working out these kinds of arguments when he came across exciting new work in what seemed a totally unrelated field: population ecology. Ecologists in the early 1970s were exploring several new approaches. “That had a big impact on me,” Hannan says. With John Freeman of the University of California, Berkeley, he wrote the seminal paper “The Population Ecology of Organizations.”

The research tradition’s name tends to provoke suspicion. Business people might say that it’s a jungle out there, but surely serious scholars shouldn’t confuse man-made organizations with the world of plants and animals. The academics stress that, indeed, their approach is not about turning organizational studies into a natural science. “Ecology was used as a source of inspiration,” Carroll says, “but not as a source for understanding organizations.” The ecological twist to organizational studies includes an emphasis on the fact that each organization’s environment is made up of other organizations. It also shifts the analysis up from a single organization to the level of whole populations of organizations.

Most industry studies would define the relevant players as companies that have opened for business. This doesn’t satisfy the organizational ecologists. They note that before an organization is formally launched, the founders have to work hard developing their plans and assembling resources. Some of these activities culminate in successful births, but others are aborted or stillborn. Studies that begin at the organization’s legal incorporation or commencement of production thus underestimate mortality rates and the degree of difficulty involved in entrepreneurial activity.

The automobile industry is a case in point. In a 1994 study, Carroll, Hannan, and their collaborators looked at producers and preproducers—defined as firms that had begun some form of organizing effort but had not reached production—for almost a century since 1886. They found that historians and economists have tended to overlook the astonishing number of hopeful producers, especially in the industry’s early years. They count 3,845 preproduction organizing attempts in the United States, of which only 11 percent succeeded in transitioning to the production stage.

Collecting data of this kind is no walk in the park. It usually involves hundreds of hours of interviews and archival research. Tracking down members of a population and recording their characteristics requires the tenacity of a detective on a chase and the attention to detail of an archaeologist on a dig. Clues come from unlikely sources. For the auto industry study, researchers relied partly on the 1,568-page Standard Catalog of American Cars 1805–1942 published for vintage car hobbyists and collectors. Carroll’s study on the microbrewery movement drew on collectors of beer mats.

Organizational ecologists suggest that the failure to appreciate the full diversity of organizations within an industry may translate into weak policy. “For example, in current policy debates concerning the competitiveness of a nation’s firms in the international marketplace, this issue often gets analyzed on the basis of a few anecdotes or highly publicized cases,” Carroll and Hannan write in their book, The Demography of Corporations and Industries. Similarly, discussions about the ability of Western nations and Japan to meet the pension burden of baby-boomers tend to focus on national social security systems and large old corporations, when in fact the greatest needs are generated by small new firms lacking private pension coverage.

Today, there is an international network of some 100 scholars working through an organizational ecology perspective. The approach has merited an entry in the forthcoming International Encyclopaedia of the Social Sciences (2002) and is recognized as a core specialty in organizational studies textbooks. A growing number of industries have been scrutinized through this lens, ranging from disk drive manufacturing to wineries and from airlines to auditing.

The Nature of Competition

One of the main insights of organizational ecology is that the environment of a firm is made up largely of other firms. Scholars in this tradition therefore have looked closely at how organizations affect each other. A key theory put forward in the original paper by Hannan and Freeman is “density dependence,” which says that organizations’ vital rates—their founding rates, growth, and mortality—depend on the total number of organizations within the relevant population. Population density is said to have two separate effects: through legitimation and through competition.

Legitimation is the process by which a certain way of doing things comes to be seen as natural or taken for granted. Legitimation increases founding rates and reduces mortality rates. Competition arises when organizations need to rely on the same pool of resources, such as capital and customers. Competition has the opposite effect of legitimation: It reduces founding rates and raises mortality rates.

Rising population density increases both legitimation and competition. However, the force of legitimation is stronger when the population density is rising from a low base, such as in the early history of an industry. The competition effect is stronger at higher densities. Combining both effects, the theory predicts that founding rates will show an inverted U-shape relationship with density, first rising as legitimation increases, then falling as competition kicks in. For the same reason, mortality rates should show a U-shape pattern, falling at first, then rising. The basic tenets of density dependence theory have been widely accepted and demonstrated to apply in many contexts.

A newer take on the organizational environment is the “Red Queen” theory, which highlights the relative nature of progress. The theory is borrowed from ecology’s Red Queen hypothesis that successful adaptation in one species is tantamount to a worsening environment for others, which must adapt in turn to cope with the new conditions. The theory’s name is inspired by the character in Lewis Carroll’s Through the Looking Glass who seems to be running but is staying on the same spot. In a 1996 paper, William Barnett describes Red Queen competition among organizations as a process of mutual learning. A company is forced by direct competition to improve its performance, in turn increasing the pressure on its rivals, thus creating a virtuous circle of learning and competition.

Barnett and David McKendrick of the University of California, San Diego, have tested this theory against data on the global hard disk drive industry. Tracking more than 150 firms over four decades, they found that those with a history of enduring competition had a higher chance of survival than those that avoided competition by technological or geographic differentiation. In line with Red Queen theory, it appears that isolation from competition, while having short-term advantages, deprives an organization of the long-term benefit of an ecology of learning, thus stymieing innovation. Their study also suggests that a lack of domestic competitive experience can prove to be a critical disadvantage when a firm is thrust into global competition.

Aging and Adapting

Organizational ecology has helped to illuminate what happens to industries over time. Many other studies have suggested that young organizations suffer a so-called “liability of newness” and have a higher risk of failure than old ones. Research in organizational ecology challenges this conclusion. Its more comprehensive data suggest that what seems like the effect of age might really be an effect of size: Infant companies may be vulnerable because they are small, not because they are young. When size is taken into account, the liability of newness often is canceled out by the liability of obsolescence.

One of the key challenges that organizations face as they age is the need to adapt to changing circumstances. Hannan’s original hunch continues to be borne out by organizational ecology research: Change is easier said than done. The social and economic environment at the time of an organization’s founding can have an enduring impact on its mission, structure, and operation. Those that try to transform core elements of their structure often experience increased risk of failure.

The scholars do not claim that organizational change is always dangerous. In the context of dramatic environmental shift, it may be necessary and beneficial to change core organizational features. However, in most cases the process of change itself can be so disruptive in the short term that the organization never gets to see the long-term benefit.

The opposing view, that organizational change is helpful and simple, may arise from case studies of successful organizations—the kind found in popular management books. These star firms may have undergone successful transformations, but their experience is not representative of the vast population of firms. “It is tempting—and many analysts succumb—to infer from this information that, had other organizations attempted the same changes, they too would have experienced success. Unfortunately, this inference comes from considering data that are heavily biased toward the successful firms,” Carroll and Hannan write.

Of course, an enduring industry, taken as a whole, can be seen to adapt to its changing environment. But organizational ecology holds that the driving mechanism for an industry’s evolution is unlikely to be the adaptation of its individual firms. Instead, it is through the selective replacement of outdated organizations that industries adapt. In the airline industry, for example, the once-dominant Pan Am, TWA, and Eastern are all no more. Old household names in retailing such as Montgomery Ward, Sears, and J.C. Penney have given way to Wal-Mart and Target. And steel giants such as Bethlehem and U.S. Steel have lost out to mini-mills such as Nucor. Thus, companies die while industries evolve. One of the current forays in this research direction is the Stanford Project on Emerging Companies, or SPEC, which Hannan directs together with James Baron, the Walter Kenneth Kilpatrick Professor of Organizational Behavior and Human Resources.

The project tracked the evolution of nearly 200 high-technology startups in Silicon Valley between 1980 and 1996, and was later extended to mid-2001, creating probably the most comprehensive database on the histories, structures, and human resource practices of this global center of entrepreneurship.

The study found several different basic models for employment relations. The most common was what the researchers call the “engineering” model, which involves selecting staff based on specific task abilities, using challenging work as the basis for employee attachment to the firm, and controlling and coordinating employee effort through peer groups. Some firms later transitioned to a “bureaucracy” model, in which control becomes more formalized.

The researchers have found that—in line with organizational ecology’s theories about the disruptive effects of change—companies that reorganized their human resource blueprints tended to suffer higher employee turnover and diminished performance. Enterprises in which the blueprint changed were more than twice as likely to fail as similar firms with blueprints that were stable. Over a three-year period, the latter firms grew at almost triple the rate of the former.

Personnel changes at the top are not disruptive as such. It is when chief executives change employment relationships that staff turnover increases. In fact, changing the blueprint seems to be most disruptive when it is implemented by the company’s first CEO, who then stays on. This could be because the founder CEO’s continued presence serves as a reminder that the organization has deviated from its original model.

The findings are especially significant, the scholars note, because it is hard to imagine a setting in which constant flux is more prevalent and where organizational change seems more justified than Silicon Valley. Given the benefits of staying the course, the authors recommend that entrepreneurs should pay more attention to picking an appropriate organizational model from the start. The study challenges the view—popular in the heyday of Silicon Valley’s “built to flip” ethos—that steady organization-building is passé in a new economy flying at Internet speed.

They add that the initial blueprint at founding should be a compromise between the current and the expected future needs—a point that few company founders seem to care about. “It’s by no means uncommon to see a founder spend more time and energy fretting about the scalability of the phone system or IT platform than about the scalability of the culture and practices for managing employees,” write Hannan and Baron.

A quarter-century since the publication of his seminal paper, Hannan says his own thinking has evolved. As one looks more closely at what exactly these organizational forms are that make up a population, it is clear that they are not merely manifestations of formal technical features. “It’s clear that they are social constructions, rooted in identities, and at some level as cultural as you can get,” he says.

Hannan does not claim to be setting the agenda for organizational ecology. The scholars in this area draw on certain shared analytical ideas but, like the industries they study, organizational ecology today is marked by theoretical diversity. This is good news, Hannan says: “It assures that talented young people still join it.” It may not be long, therefore, before they start calling him a great-grandfather.

Old Assumptions Challenged

Organizations change easily and often. Actually, the research shows inertia reigns. The relevant players in an industry are companies that have opened for business. Industries are also shaped by ideas from companies that are aborted or stillborn. Companies without competitors have the best chance of survival. Companies with enduring competition are better survivors. New organizations are the most likely to fail. It is not their youth but their small size that is the biggest risk factor. Personnel changes at the top of a company are disruptive. It is more disruptive for existing executives to change the company blueprint.