The biennial “Chaos Report” on IT project success from The Standish Group reports that the success/failure ratio flipped between 1994 and 2006. In 1994 the ratio for “flat failures” vs. “complete successes” was a depressing 31% vs. 16%; in 2006 it was a more encouraging 19% vs. 35%. (The work is reported in CIO; the Standish Group web site is remarkably sullen, and doesn’t seem to have any press releases, let alone publicly available recent data.)
On page 2 of the CIO story, the Standish CEO says: “Seventy-three percent of projects with labor cost of less than $750,000 succeed. . . . But only 3 percent of projects a with labor cost of over $10 million succeed. I would venture to say the 3 percent that succeed succeeded because they overestimated their budget, not because they were managed properly.” A $750,000 project is pretty tiny: six developers for six months, at $250,000/developer/year fully loaded. Even a $10 million project is 20 developers for two years.
This result matches received wisdom that large projects are more likely to fail, which I attribute at least in part to the cognitive challenge of wrapping one’s head around large problems.
What should one do about it? It implies that smaller projects are the only way to go – but what if one has ambitious goals? If it’s true that one can construct complex solutions out of many small, simple parts, everything’s fine. But I’m deeply suspicious of the “divide and conquer” or “linearization” assumption. There are many important problems that just can’t be broken up, from inverting a matrix to simulating non-linear systems.
This may be a cultural reality check: many ambitious goals may simply not be achievable. Humility may be the best way to ensure success. I doubt politicians and business executives want to hear this. Trying to fly too high brought Icarus down – exactly as his engineer-father Daedalus had warned.
And things may not get better: as technology progresses, the complexities of our systems will grow, and linear solutions become even less useful. As the interconnectedness and intangibility of society grows, we may have to become more humble, not more bold, because that will be the only way to get stuff done. It’s counter-intuitive that as technology progresses we need to become less, not more, ambitious, but that may be the way things work with the new intangibles.
NOTES
My thanks to Henry Yuen for referring this story.
I have some reservations about the Standish data. It’s proprietary, and there are academics who’ve questioned it for years. CIO provides some background on Chaos Report and its methods in an interview with the CEO; it also summarizes questions about their method. One has to wonder how the sample population has changed over the years. If the number of small projects in the sample has grown over time, then success reported above would increase simply because smaller projects fail less often, not because project management performance has improved.
Sunday, July 22, 2007
Tuesday, July 17, 2007
Business: a City, not an Ecosystem
Geoffrey West’s work on scaling in cities provides ammunition for my critique of the “business ecosystem” analogy (Ecosystem alert, Eco mumbo jumbo). New Scientist reports on a recent paper by West and co-workers which found that some urbanization processes differ dramatically from biological ones (references below).
Describing the city as an organism is a much-loved metaphor; New Scientist quotes Frank Lloyd Wright waxing lyrical about “thousands of acres of cellular tissue . . . enmeshed by an intricate network of veins and arteries.”
We like to think that cities work like biological entities, just as we like to think that industries work like networks of organisms. But West & Co’s work indicates that the analogy is flawed. As animals get larger, their metabolism slows down. This is true in some respects for cities, but in others the opposite is true. Infrastructure metrics, like the numbers of gas stations and miles of paved roads scale like biological ones: the amount grows less slowly than the size of the city. But for the things that really count, things speed up. For example, measures of wealth creation and innovation - the number of patents, total wages, GDP - grow more rapidly with size. Bigger cities have a faster metabolism than smaller ones, unlike animals.
Industries resemble cities more than they resemble ecosystems. Increasing returns with size and non-zero sums are key characteristics that are found in both cities and industries, but not biological systems. “Business is Urbanism” is a more accurate and productive metaphor than “Business is an Ecosystem”.
P.S. While we’re talking about ecosystems... The very notion of ecosystem is, of course, itself a metaphor: Nature is a System. The American Heritage Dictionary defines a system as “A group of interacting, interrelated, or interdependent elements forming a complex whole.” It is presumed that there is an observable whole; that it can be broken down into elements; and that the elements interact. So when people use the Business is an Ecosystem metaphor, I think what they’re really doing is simply using Business is a System, and cloaking it with the numinous mantle of Nature (cf. Ecosystem alert).
References
Dana Mackenzie, Ideas: the lifeblood of cities, New Scientist, 23 May 2007 (subscription wall)
Bettencourt, Lobo, Helbing, Kuhnert & West, Growth, innovation, scaling, and the pace of life in cities, Proceedings of the National Academy of Sciences, vol 104, p 7301, 24 April 2007
Describing the city as an organism is a much-loved metaphor; New Scientist quotes Frank Lloyd Wright waxing lyrical about “thousands of acres of cellular tissue . . . enmeshed by an intricate network of veins and arteries.”
We like to think that cities work like biological entities, just as we like to think that industries work like networks of organisms. But West & Co’s work indicates that the analogy is flawed. As animals get larger, their metabolism slows down. This is true in some respects for cities, but in others the opposite is true. Infrastructure metrics, like the numbers of gas stations and miles of paved roads scale like biological ones: the amount grows less slowly than the size of the city. But for the things that really count, things speed up. For example, measures of wealth creation and innovation - the number of patents, total wages, GDP - grow more rapidly with size. Bigger cities have a faster metabolism than smaller ones, unlike animals.
Industries resemble cities more than they resemble ecosystems. Increasing returns with size and non-zero sums are key characteristics that are found in both cities and industries, but not biological systems. “Business is Urbanism” is a more accurate and productive metaphor than “Business is an Ecosystem”.
P.S. While we’re talking about ecosystems... The very notion of ecosystem is, of course, itself a metaphor: Nature is a System. The American Heritage Dictionary defines a system as “A group of interacting, interrelated, or interdependent elements forming a complex whole.” It is presumed that there is an observable whole; that it can be broken down into elements; and that the elements interact. So when people use the Business is an Ecosystem metaphor, I think what they’re really doing is simply using Business is a System, and cloaking it with the numinous mantle of Nature (cf. Ecosystem alert).
References
Dana Mackenzie, Ideas: the lifeblood of cities, New Scientist, 23 May 2007 (subscription wall)
Bettencourt, Lobo, Helbing, Kuhnert & West, Growth, innovation, scaling, and the pace of life in cities, Proceedings of the National Academy of Sciences, vol 104, p 7301, 24 April 2007
Sunday, July 15, 2007
When physicists see a power law, they think in terms of phase transitions, and they smell Nobel prizes. They are like sharks with blood in the water
--- Steven Strogatz on the fuss about scaling laws, quoted in New Scientist, Ideas: the lifeblood of cities, 23 May 2007
In context, from New Scientist:
During the 20th century, many researchers studying urban growth focused on economies of scale and their effect on wages. In 1974, Vernon Henderson of Brown University in Rhode Island proposed that cities reach an optimal size by growing until their workers' per capita income reaches a maximum; when it starts to decline, workers leave for other cities. More recently, researchers including West have tried to identify deeper mechanisms behind these societal patterns. Though West is a physicist by training, his reputation stems mostly from his pioneering and controversial work on scaling laws in biology - how things change with size.
What is all the fuss about scaling laws? "Physicists are used to thinking about extremely large systems of identical particles," says Steven Strogatz, a mathematician at Cornell University in Ithaca, New York. Take a piece of iron: at high temperatures, the spins of the particles jiggle around in random directions. If you gradually lower the temperature, the spins stay random until you reach a critical point - then they suddenly line up, and you have a ferromagnet.
This switch from disorder to order is called a phase transition. In the 1960s, physicists noticed that phase transitions follow certain universal patterns, called power laws, even if they have nothing in common physically. Kenneth Wilson of Cornell showed in the 1970s that these power laws come about through the growth of fractal structures, work which won him the Nobel prize in 1982. Since then, Strogatz says, "When physicists see a power law, they think in terms of phase transitions, and they smell Nobel prizes. They are like sharks with blood in the water."
In context, from New Scientist:
During the 20th century, many researchers studying urban growth focused on economies of scale and their effect on wages. In 1974, Vernon Henderson of Brown University in Rhode Island proposed that cities reach an optimal size by growing until their workers' per capita income reaches a maximum; when it starts to decline, workers leave for other cities. More recently, researchers including West have tried to identify deeper mechanisms behind these societal patterns. Though West is a physicist by training, his reputation stems mostly from his pioneering and controversial work on scaling laws in biology - how things change with size.
What is all the fuss about scaling laws? "Physicists are used to thinking about extremely large systems of identical particles," says Steven Strogatz, a mathematician at Cornell University in Ithaca, New York. Take a piece of iron: at high temperatures, the spins of the particles jiggle around in random directions. If you gradually lower the temperature, the spins stay random until you reach a critical point - then they suddenly line up, and you have a ferromagnet.
This switch from disorder to order is called a phase transition. In the 1960s, physicists noticed that phase transitions follow certain universal patterns, called power laws, even if they have nothing in common physically. Kenneth Wilson of Cornell showed in the 1970s that these power laws come about through the growth of fractal structures, work which won him the Nobel prize in 1982. Since then, Strogatz says, "When physicists see a power law, they think in terms of phase transitions, and they smell Nobel prizes. They are like sharks with blood in the water."
Not that weird
Peter Pitsch’s The Innovation Age (1996) made me question something I’ve taken for granted: that complexity and uncertainty in the economy is growing, and doing so at an unprecedented rate. Pitsch’s book is based on this premise, and it made me wonder: what is the evidence?
The number of industry players who are inter-connected may be growing due to the Internet and cheap global travel, but an individual companies is not necessarily directly connected to more counterparts. It’s a bigger graph, but when one looks at individual nodes, the connectivity is much as it has always been.
Uncertainty isn’t new, either. Pitsch mentions the late Middle Ages as a tumultuous period that produced amazing innovation, and the Industrial Revolution was similar. The uncertainty in aggregate may be larger today, but so is the world population; has the normalized per-capita uncertainty grown? A reasonable measure might be stock market volatility. Schwert’s data for the 1859 – 1987 period doesn’t show any trends I can see with the naked eye (G.W. Schwert, “Why Does Stock Market Volatility Change Over Time," Journal of Finance, vol. 44, pp. 1115-1153, 1989). Market uncertainty, at least, is much the same.
I now believe that this is the Special Present Fallacy at play again. We’re always biased to see the present moment as exceptional; the odds are that it’s not.
The number of industry players who are inter-connected may be growing due to the Internet and cheap global travel, but an individual companies is not necessarily directly connected to more counterparts. It’s a bigger graph, but when one looks at individual nodes, the connectivity is much as it has always been.
Uncertainty isn’t new, either. Pitsch mentions the late Middle Ages as a tumultuous period that produced amazing innovation, and the Industrial Revolution was similar. The uncertainty in aggregate may be larger today, but so is the world population; has the normalized per-capita uncertainty grown? A reasonable measure might be stock market volatility. Schwert’s data for the 1859 – 1987 period doesn’t show any trends I can see with the naked eye (G.W. Schwert, “Why Does Stock Market Volatility Change Over Time," Journal of Finance, vol. 44, pp. 1115-1153, 1989). Market uncertainty, at least, is much the same.
I now believe that this is the Special Present Fallacy at play again. We’re always biased to see the present moment as exceptional; the odds are that it’s not.
Thursday, July 12, 2007
Factoids: cost of clinical trials
According to Thomson CenterWatch, a publishing company that focuses on the clinical trials industry, companies need to recruit about 4,000 people to test an experimental drug at a cost of up to $25,000 for each person. That translates into $100 million at the high end.
Source: Advertorial on Clinical Trials in New Scientist, 16 June 2007
Source: Advertorial on Clinical Trials in New Scientist, 16 June 2007
Factoid: A typical cellphone user spends 80% of his or her time communicating with just four other people
--- Source: Stefana Broadbent, an anthropologist who leads the User Adoption Lab at Swisscom, cited by the Economist in Tech Quaterly story on June 9, 2007: Home truths about telecoms.
They also quote her thus: "The most fascinating discovery I've made this year is a flattening in voice communication and an increase in written channels. . . . Users are showing a growing preference for semi-synchronous writing over synchronous voice." The Economist's gloss: "Her research in Switzerland and France found that even when people are given unlimited cheap or free calls, the number and length of calls does not increase significantly. This may be because there is only so much time you can spend talking; and when you are on the phone it is harder to do other things. Written channels such as e-mail, text-messaging and IM, by contrast, are discreet and allow contact to be continuous during the day."
It seems writing really is a useful alternative channel. I guess there's a reason why the Blackberry was so succesful.
They also quote her thus: "The most fascinating discovery I've made this year is a flattening in voice communication and an increase in written channels. . . . Users are showing a growing preference for semi-synchronous writing over synchronous voice." The Economist's gloss: "Her research in Switzerland and France found that even when people are given unlimited cheap or free calls, the number and length of calls does not increase significantly. This may be because there is only so much time you can spend talking; and when you are on the phone it is harder to do other things. Written channels such as e-mail, text-messaging and IM, by contrast, are discreet and allow contact to be continuous during the day."
It seems writing really is a useful alternative channel. I guess there's a reason why the Blackberry was so succesful.
Sunday, July 08, 2007
Ecosystem alert
When you see references to ecosystems in a business story, raise the shields. Someone is trying to mess with your mind.
The current Business Week has two good examples. An adulatory story about the "Apple ecosystem" (Welcome to Planet Apple, which ran as Welcome to Apple World in hard copy) describes how the company has built its network of partners. Implicit is Iansiti and Levien's notion that the most influential companies are "keystone species in an ecosystem." As I argued in Eco mumbo jumbo, the analogy is flawed in a long list of ways. For example: species don't choose to be keystones; companies interact vountarily, but one organism consumes another against its will; and biological systems have neither goals nor external regulators, whereas industries have both.
The ecosystem analogy is used unthinkingly in this story, judging by the hodgpodge of other metaphors that are used: "[the] ecosystem has morphed from a sad little high-tech shtetl into a global empire," "[its] new flock of partners," "a gated, elitist community," "the insular world of the Mac," "the Apple orchard . . . is still no Eden." Note, though, that most of them refer to places, with a nod to nature.
To get a sense of what's really going on when the ecosystem metaphor is used, let's look at another story, Look Who's Fighting Patent Reform. Computing companies have been pushing for patent reform on Capitol Hill, but "[t]he past few weeks have brought an unexpected surge of opposition from what one lobbyist calls the 'innovation ecosystem'—a sprawling network of entrepreneurs, venture capitalists, trade groups, drug and medical equipment manufacturers, engineering societies, and research universities." It's a term used by the special pleader. The only substantive resemblance to an ecosystem is that these groups connect to each other in network. The rhetorical benefit, though, is to invoke the commonplace Nature Is Good. Nature is unspoiled, bountiful, self-regulating: the antithesis of concrete-covered recklessly-regulating partisan politicking. Nature is a metaphor that appeals to both sides of the political divide: it's organic, but competitive; it's an inter-related, but dynamic; it's nurturing, but stern in its consequences. It's therefore ideal when trying to put a halo around an otherwise unsympathetic subject.
The current Business Week has two good examples. An adulatory story about the "Apple ecosystem" (Welcome to Planet Apple, which ran as Welcome to Apple World in hard copy) describes how the company has built its network of partners. Implicit is Iansiti and Levien's notion that the most influential companies are "keystone species in an ecosystem." As I argued in Eco mumbo jumbo, the analogy is flawed in a long list of ways. For example: species don't choose to be keystones; companies interact vountarily, but one organism consumes another against its will; and biological systems have neither goals nor external regulators, whereas industries have both.
The ecosystem analogy is used unthinkingly in this story, judging by the hodgpodge of other metaphors that are used: "[the] ecosystem has morphed from a sad little high-tech shtetl into a global empire," "[its] new flock of partners," "a gated, elitist community," "the insular world of the Mac," "the Apple orchard . . . is still no Eden." Note, though, that most of them refer to places, with a nod to nature.
To get a sense of what's really going on when the ecosystem metaphor is used, let's look at another story, Look Who's Fighting Patent Reform. Computing companies have been pushing for patent reform on Capitol Hill, but "[t]he past few weeks have brought an unexpected surge of opposition from what one lobbyist calls the 'innovation ecosystem'—a sprawling network of entrepreneurs, venture capitalists, trade groups, drug and medical equipment manufacturers, engineering societies, and research universities." It's a term used by the special pleader. The only substantive resemblance to an ecosystem is that these groups connect to each other in network. The rhetorical benefit, though, is to invoke the commonplace Nature Is Good. Nature is unspoiled, bountiful, self-regulating: the antithesis of concrete-covered recklessly-regulating partisan politicking. Nature is a metaphor that appeals to both sides of the political divide: it's organic, but competitive; it's an inter-related, but dynamic; it's nurturing, but stern in its consequences. It's therefore ideal when trying to put a halo around an otherwise unsympathetic subject.
Friday, July 06, 2007
Trading on News
Follow the money, if you want to know where the action is in AI (and most other things). Trading houses are buying tagged news feeds so that they can process them as input for algorithmic trading. That'sll have to be a pretty smart news reader.
The Economist story that reports this development contains these factoids:
Taken a step further, I can imagine the trading houses selling re-tagged news feeds back to Dow Jones and Reuters. Suitably anonimized, aggregated, and delayed to protect the individual movers, information on which news items triggered trades would be useful at second order. And then the news providers can sell the re-re-tagged feeds to the traders, who'll then sell back the re-re-re-tagged . . .
The Economist story that reports this development contains these factoids:
- Algorithmic trading accounts for a third of all share trades in America
- The Aite Group reckons it will make up more than half the share volumes and a fifth of options trades by 2010
- The new London Stock Exchange system catering to the growth of algorithmic trading cuts trading times down to ten milliseconds; on its first day, it processed up to 1,500 orders a second, compared with 600 using its previous system
Taken a step further, I can imagine the trading houses selling re-tagged news feeds back to Dow Jones and Reuters. Suitably anonimized, aggregated, and delayed to protect the individual movers, information on which news items triggered trades would be useful at second order. And then the news providers can sell the re-re-tagged feeds to the traders, who'll then sell back the re-re-re-tagged . . .
Tuesday, July 03, 2007
Factoid: Americans spent only 0.2% of their money but 10% of their time on the internet
Source: Austan Goolsbee and Peter Klewnow, "Valuing Consumer Products by the Time Spent Using Them: An Application to the Internet," draft available at http://faculty.chicagogsb.edu/austan.goolsbee/research/timeuse.pdf. This result suggests that conventional consumer surplus calculations significantly understate the value of the internet.
Paper abstract:
Paper abstract:
For some goods, the main cost of buying the product is not the price but rather the time it takes to use them. Only about 0.2% of consumer spending in the U.S., for example, went for Internet access in 2004 yet time use data indicates that people spent around 10% of their entire leisure time going online. For goods like that, estimating price elasticities with expenditure data can be difficult and, therefore, estimated welfare gains highly uncertain. We show that for time-intensive goods like the Internet, a simple model in which both expenditure and time contribute to consumption can be used to estimate the consumer gains to a good using just the data on time use and the opportunity cost of people's time (i.e., the wage). The theory predicts that higher wage internet subscribers should spend less time online (for non-work reasons) and the degree to which that is true determines the elasticity of demand. Based on expenditure and time use data and our elasticity estimate, we calculate that consumer surplus from the Internet may be around 2% of full-income, or several thousand dollars. This is an order of magnitude larger than what one obtains from a back-of-the-envelope calculation using data from expenditures.
Monday, July 02, 2007
The Known, the Unknown, the Unknowable
Thanks to a acknowledgement in Nassim Taleb's The Black Swan, I've found this fascinating program on The Known, Unknown, and Unknowable run by Jesse Ausubel of the Sloan Foundation.
Gomory outlines this vision in a short essay, published in Scientific American in 1995. He believes that the artificial is simpler, and thus more predictable, than the natural. However, he notes: "Large pieces of software, as they are expanded and amended, can develop a degree of complexity reminiscent of natural objects, and they can and do behave in disturbing and unpredictable ways."
The program led to a workshop at Columbia in 2000; I look forward to digging in to the proceedings. One of the papers that caught my eye was Ecosystems and the Biosphere as Complex Adaptive Systems by Simon Levin. It seems an open question whether evolution increases the resiliency of ecosystems or leads to criticality - a very important matter for business people hoping that aping ecosystems will improve stability!
Gomory outlines this vision in a short essay, published in Scientific American in 1995. He believes that the artificial is simpler, and thus more predictable, than the natural. However, he notes: "Large pieces of software, as they are expanded and amended, can develop a degree of complexity reminiscent of natural objects, and they can and do behave in disturbing and unpredictable ways."
The program led to a workshop at Columbia in 2000; I look forward to digging in to the proceedings. One of the papers that caught my eye was Ecosystems and the Biosphere as Complex Adaptive Systems by Simon Levin. It seems an open question whether evolution increases the resiliency of ecosystems or leads to criticality - a very important matter for business people hoping that aping ecosystems will improve stability!
Subscribe to:
Posts (Atom)