I was honored to participate in a panel in DC on "An FCC for the Internet Age: Reform and Standard-Setting" organized by Silicon Flatirons, ITIF and Public Knowledge on March 5th, 2010. My introductory comments tried to summary the "resilience principles" in five minutes: the video is available on the Public Knowledge event page, starting at time code 02:04:45. The panel starts at around 01:57:00.
The earlier, fifteen minute pitch I gave on a panel on "The Governance Challenges of Cooperation in the Internet Ecosystem" at the Silicon Flatirons annual conference in Boulder on February 1st, 2010 can be found here at time code 01:36:00. My slides are up on Slideshare.net, and a paper is in preparation for JTHTL.
This work is an outgrowth of my TPRC 2008 paper “Internet Governance as Forestry” (SSRN).
Showing posts with label complexity. Show all posts
Showing posts with label complexity. Show all posts
Friday, March 26, 2010
Wednesday, February 10, 2010
The internet is not an ecosystem, but…
The “internet ecosystem” metaphor is ubiquitous; I’ve used it myself, though with some trepidation. I think I can now reconcile why it’s both wrong and useful.
It’s wrong, strictly speaking, since many aspects of the ecosystem-internet mapping are questionable. As I blogged in 2007 about the “business ecosystem” terminology, the validity of the metaphor is undermined by quite a large number of mapping mismatches:
It’s wrong, strictly speaking, since many aspects of the ecosystem-internet mapping are questionable. As I blogged in 2007 about the “business ecosystem” terminology, the validity of the metaphor is undermined by quite a large number of mapping mismatches:
Number: a food web consists of billions of interactions among animals and plants; a business web comprises a relatively small number of companies
Metrics: Biomass a typical rough measure of an ecosystem; does that map to total revenue, profitability, return on investment, or something else?
Topology: An ecosystem is a lossy, one-way energy flow; as each organism is eaten by the next, energy is lost. Business relationships are reciprocal, and generate value.
Time scales: Species change slowly, but companies can change their role in a system overnight through merger, acquisition or divestiture.
Choice: Interactions between firms can be changed by contract, whereas that between species is not negotiable except perhaps over very long time scales by evolution of defensive strategies.
Foresight: Humans are pre-eminent among animals in their ability to anticipate the behavior of other actors, explore counter-factuals, think through What If scenarios, etc. The response of a system containing humans to some change is therefore much more complex than that of a human-free ecosystem. “Dumb” agents in an adaptive system respond to the change; humans respond to how they think other humans will respond to their response to those people’s responses etc.
Goals: Biological systems don’t have goals, but human ones do. There are no regulatory systems external to ecosystems in a state of nature (if such things still exist on this planet), but there are many, such as rule of law and anti-trust, in human markets. Natural processes don’t care about equity or justice, but societies do, and impose them on business systems. If ecosystems were a good model for business networks, there would be no need for anti-trust regulation.
The connotations of the metaphor are also misleading. Ecosystems are often used to connote stability and vibrant self-regulation; in fact, they often suffer catastrophic collapses. Companies are exhorted to invest in their ecosystem with the goal of becoming a keystone species. It’s not clear why they should do so, from the ecosystem perspective: keystone species don’t typically represent a lot of biomass. Their “bottleneck position”, however, is attractive from the perspective of a company that wants to extract rents through market power.
However, the ecosystem concept has gained traction because there is a deeper truth: both the internet and ecosystems are both examples of complex adaptive systems. (A complex adaptive system may be defined as a collection of interacting, adaptive agents; other examples include the immune system, the human body, stock markets, and economies. Note that adaptive systems are often nested.)
Thus, the internet is to an ecosystem as a whale is to an elephant. It could be useful to think in terms of elephants if one has to manage oceans but doesn’t know much about whales, since both are large, social mammals. However, one can just as well explain whales in terms of elephants – and the differences, e.g. living on land vs. in water – can be decisive in some cases.
With this realization, the utility and limitations of using an ecosystem metaphor when thinking about the internet, as I did in my Internet Governance as Forestry paper, have become much clearer to me. Lessons from managed ecosystems can illuminate the dynamics and pitfalls of managing the internet, and principles (such as the Resilience Principles I outlined in my recent talk at Silicon Flatirons; my presentation starts around time code 01:36:00 of the video) derived from one can be applied to the other.
However, the ecosystem concept has gained traction because there is a deeper truth: both the internet and ecosystems are both examples of complex adaptive systems. (A complex adaptive system may be defined as a collection of interacting, adaptive agents; other examples include the immune system, the human body, stock markets, and economies. Note that adaptive systems are often nested.)
Thus, the internet is to an ecosystem as a whale is to an elephant. It could be useful to think in terms of elephants if one has to manage oceans but doesn’t know much about whales, since both are large, social mammals. However, one can just as well explain whales in terms of elephants – and the differences, e.g. living on land vs. in water – can be decisive in some cases.
With this realization, the utility and limitations of using an ecosystem metaphor when thinking about the internet, as I did in my Internet Governance as Forestry paper, have become much clearer to me. Lessons from managed ecosystems can illuminate the dynamics and pitfalls of managing the internet, and principles (such as the Resilience Principles I outlined in my recent talk at Silicon Flatirons; my presentation starts around time code 01:36:00 of the video) derived from one can be applied to the other.
Monday, February 08, 2010
Resilience and Realpolitik
Resilience is a fashionable meme - rightly so, since it offers an alternative to the "find the efficient optimum" approach to solving problems in political economy. (I would say so, of course; see e.g. my presentation at Silicon Flatirons recently, and my paper on forestry as a metaphor for internet governance.)
As reported by The Economist (A needier era: The politics of global disruption, and how they may change, Jan 28th 2010), a report for the Brookings Institution on international politics in an age of want suggests that Governments should think more in terms of reducing risk and increasing resilience to shocks than about boosting sovereign power. This is analogous to advocating reducing risk and increasing resilience vs. boosting wealth creation in the economy. The reason given is that the new threats are networks (of states and non-state actors) and unintended consequences (of global flows of finance, technology and so on).
I've seen (and propagated) the same memes in the context technology policy: the determining factors are inter-locking networks of agents, and unintended consequences that shift more quickly than legislation.
It's ironic, given my claim that the resilience approach is a counter to neoclassical economics, that the article closes with a Milton Friedman quote...
As reported by The Economist (A needier era: The politics of global disruption, and how they may change, Jan 28th 2010), a report for the Brookings Institution on international politics in an age of want suggests that Governments should think more in terms of reducing risk and increasing resilience to shocks than about boosting sovereign power. This is analogous to advocating reducing risk and increasing resilience vs. boosting wealth creation in the economy. The reason given is that the new threats are networks (of states and non-state actors) and unintended consequences (of global flows of finance, technology and so on).
I've seen (and propagated) the same memes in the context technology policy: the determining factors are inter-locking networks of agents, and unintended consequences that shift more quickly than legislation.
It's ironic, given my claim that the resilience approach is a counter to neoclassical economics, that the article closes with a Milton Friedman quote...
Monday, November 02, 2009
Another problem with the ecosystem metaphor
Lay commentators - including yours truly - have a weakness for using the biological communities as a metaphor for social interactions, particularly business competition. No matter the litany of quibbles (see my own at Eco mumbo jumbo); it's just so tempting to talk about a business ecosystem!
A recent News Focus piece in Science (Erik Stokstad, On the Origin of Ecological Structure, Science 2 October 2009, Vol. 326. no. 5949, pp. 33 - 35; there's also an interview with Stokstad in the podcast of 2 October, 2009) raises a more fundamental problem with the utility of the metaphor: ecologists themselves are still struggling to understand what dictates the kinds and proportions of organisms in communities ranging from meadows to montane forests.
Stokstad writes that "there is still no consensus on the relative importance of the various forces [that influence community formation, like competition, predation, and disturbance]. Darwin and many later ecologists emphasized competition among species, but proponents of a controversial theory of biodiversity that assumes competition has no impact argue that immigration and other random demographic events can account for much of the apparent makeup of communities. As a result, ecologists have a long way to go to come up with formulas that predict how communities might arise and change."
If ecologists can't explain community dynamics in biology, it's dangerous to make inferences by analogy about the influence of (say) competition and disturbance in commercial systems. Which is a pity, since that's just what I've tried to do myself...
A recent News Focus piece in Science (Erik Stokstad, On the Origin of Ecological Structure, Science 2 October 2009, Vol. 326. no. 5949, pp. 33 - 35; there's also an interview with Stokstad in the podcast of 2 October, 2009) raises a more fundamental problem with the utility of the metaphor: ecologists themselves are still struggling to understand what dictates the kinds and proportions of organisms in communities ranging from meadows to montane forests.
Stokstad writes that "there is still no consensus on the relative importance of the various forces [that influence community formation, like competition, predation, and disturbance]. Darwin and many later ecologists emphasized competition among species, but proponents of a controversial theory of biodiversity that assumes competition has no impact argue that immigration and other random demographic events can account for much of the apparent makeup of communities. As a result, ecologists have a long way to go to come up with formulas that predict how communities might arise and change."
If ecologists can't explain community dynamics in biology, it's dangerous to make inferences by analogy about the influence of (say) competition and disturbance in commercial systems. Which is a pity, since that's just what I've tried to do myself...
Thursday, April 30, 2009
The “business ecosystem” subtext
A feature in the new Christian Science Monitor on restoring the Galápagos to their state before the invasives arrived – rats, dogs, lizards, and especially humans – suddenly revealed to me why the “business ecosystem” rhetoric makes me uneasy.
Here’s the key passage:
Catastrophe is as much a part of complex system behavior as continuity, but it's something we'd rather not think about too much. Radical change is bad news for incumbents - and it's bad for all of us when the "incumbents" are rare plant and animal species on the edge of extinction - but it is good news for newcomers trying to make their mark and change the world.
P.S. Here’s the quote in a little more context:
Here’s the key passage:
“If an ecosystem is a community of life forms that have evolved together and achieved equilibrium, then the restoration of that ecosystem begins with the removal of everything that upsets the balance.”Users of the “business ecosystem” metaphor perhaps aren't even conscious that their goal is equilibrium, but I’ve now realized that it’s a foundation of this world view. Everybody needs stability in their lives, even when they also crave novelty; this is particularly true for large technology companies.
Catastrophe is as much a part of complex system behavior as continuity, but it's something we'd rather not think about too much. Radical change is bad news for incumbents - and it's bad for all of us when the "incumbents" are rare plant and animal species on the edge of extinction - but it is good news for newcomers trying to make their mark and change the world.
P.S. Here’s the quote in a little more context:
Certainly, reconstructing nature is a prospect fraught with contradictions. Can it really be natural if it is created by human design?
Cruz and fellow conservationists operate on a simple formula: If an ecosystem is a community of life forms that have evolved together and achieved equilibrium, then the restoration of that ecosystem begins with the removal of everything that upsets the balance. And so, somewhat paradoxically, the conservation of Galápagan ecosystems inevitably starts with a meticulous campaign of eradication. Animals introduced by people must go. Once the slate is wiped clean, native species, some of which continue to exist only in captivity – like Lonesome George, the iconic giant tortoise who's the last of his breed – can be reintroduced. Then the community, a system of checks and balances honed to perfection over time – of grazing tortoises and plants, birds and seeds that need each other – can reestablish.
Wednesday, December 10, 2008
Reforming the FCC: Etiology
In Reforming the FCC: Diagnosis, I argued that the FCC must these days supervise a much more complex and adaptive situation. This post examines the causes of this situation. I’ll consider responses in a subsequent post.
As an exercise, consider whether the following attributes apply to 21st century communications (which I’ll also refer to as ICT, for lack of a better term): the absence of a global controller; nested, hierarchical organization; dispersed interactions; never-ending novelty; constant selection among candidate solutions; and rapid adaptation to new circumstances. I believe they definitely describe the internet/web – and they are much less applicable to the silo’d world of telecommunications and analog broadcasting of only a few decades ago.
These attributes are the hallmarks of complexity and adaptive, non-linear systems. 21st Century communications is a complex adaptive social system, but the FCC was set up to manage a 20th century industry which was complicated but not complex. This is the deep reason why the institution needs to change.
The adaptive cycle
A key attribute of complex adaptive systems is that they cycle through distinct stages. I’ll describe it here using the example of ecosystems (where it was introduced) before turning to ICT.
During the growth stage, there is rapid colonization of recently disturbed areas, for example after a fire or wind storm has removed large amounts of biomass in a forest. The connectedness between organisms is low, which leads to high resilience; the loss of one species doesn’t lead to the loss of another. As the forest matures, it moves into the maturity phase of the cycle, which is dominated by the accumulation of material. The network of connections between biomass and nutrients becomes increasingly tight, and fragile; every niche in the forest is filled, and every resource is used. Organisms become much more interdependent; food chains become dense and interconnected. The maturity phase is followed by a dramatic release, triggered in a forest by fire, drought, insect pests, etc. A lot of energy is unbound, and networks are broken up. This sets the scene for the fourth phase, reorganization: opportunistic species that have been suppressed by the stable configuration of the maturity phase move in. This is a time of innovation and restructuring, laying the groundwork for a return to another growth phase.
The behavior of managed ecosystems is shaped by three properties: the accumulation of potential, the degree of connectedness between elements, and the resilience of the system in the face of shocks. The same properties apply to complex human enterprises like modern communications.
The adaptive cycle alternates periods of gradual accumulation of potential (e.g. biomass, socio-economic capital or know-how, depending on the kind of system) with sudden and often unexpected disruptions that reorganize that potential. Connectedness is high at maturity, but that is also the time when resilience to shocks is at its lowest. This cycle of aggregation followed by restructuring leads to innovation; but the release phase is often a surprise, and frequently an unpleasant one for those who were successful in the maturity phase. It is thus often experienced as a crisis.
Decision Environments
One can recognize the phases of the adaptive cycle in the internet/web, and in the larger system of communications governance. It is helpful to parse the system into four decision environments that represent different hierarchical layers:
Another hallmark of complex adaptive systems – and one of the hardest challenges for a regulator – is unexpected novelty. Changes in the state of a complex system are usually unexpected, in part because many dynamics are invisible. Surprises are particularly noticeable when they lead to release.
Here are some recent reminders that the innovation that we expect from complex systems usually comes as a surprise:
Many surprises come from contagion between problem domains that were previously considered distinct. XM/Sirius’s problems came at the intersection of personal computing devices with broadcasting; music publishing’s crisis arose from software and networking innovations that led to the P2P distribution of digital content; and the open source software movement informed Wikipedia.
Problem scope
A consequence of interlocking decision environments and intersecting problem domains is that the unit of analysis for the FCC is no longer a distinct, largely independent, well-defined industry associated with a particular technology and its own Title in the Communications Act.
Attention needs to shift from industries to problem domains, and away from solutions for a particular industry, technology and even institution or statute. For example, a policy imperative like lawful intercept is no longer limited to telephony, which leads to conflicts such as the competing definitions of information services in CALEA and the Communications Act. This is an example of the importance of the Big Picture principle for managing adaptive system. (I’ll review this principle and its three companions – Delegation, Flexibility and Diversity – in the next post.)
However, simply broadening some existing statute to cover all new possibilities is counter-productive. It conflicts with the other three principles, and falls victim to the fallacy that narrow-minded control of a single variable leads to a healthy outcome; in adaptive systems, it leads eventually to an even worse crisis.
In conclusion, the FCC is really facing a system problem, not an institutional one. Even if today’s procedural problems within the Commission were completely solved, it would not address the challenges of a qualitatively more complex and unpredictable regulation “subject”, that is, the market/culture system where innovation and growth takes place. Nor would it speak to the problems faced at the political level where the social acceleration of time poses existential challenges to the rule of law, and profoundly complicates the separation of powers between the legislature, executive, and judiciary market capitalism, and liberal democracy.
I’ll turn to the question of how the FCC should respond in the next post.
Further reading
The adaptive cycle:
As an exercise, consider whether the following attributes apply to 21st century communications (which I’ll also refer to as ICT, for lack of a better term): the absence of a global controller; nested, hierarchical organization; dispersed interactions; never-ending novelty; constant selection among candidate solutions; and rapid adaptation to new circumstances. I believe they definitely describe the internet/web – and they are much less applicable to the silo’d world of telecommunications and analog broadcasting of only a few decades ago.
These attributes are the hallmarks of complexity and adaptive, non-linear systems. 21st Century communications is a complex adaptive social system, but the FCC was set up to manage a 20th century industry which was complicated but not complex. This is the deep reason why the institution needs to change.
The adaptive cycle
A key attribute of complex adaptive systems is that they cycle through distinct stages. I’ll describe it here using the example of ecosystems (where it was introduced) before turning to ICT.
During the growth stage, there is rapid colonization of recently disturbed areas, for example after a fire or wind storm has removed large amounts of biomass in a forest. The connectedness between organisms is low, which leads to high resilience; the loss of one species doesn’t lead to the loss of another. As the forest matures, it moves into the maturity phase of the cycle, which is dominated by the accumulation of material. The network of connections between biomass and nutrients becomes increasingly tight, and fragile; every niche in the forest is filled, and every resource is used. Organisms become much more interdependent; food chains become dense and interconnected. The maturity phase is followed by a dramatic release, triggered in a forest by fire, drought, insect pests, etc. A lot of energy is unbound, and networks are broken up. This sets the scene for the fourth phase, reorganization: opportunistic species that have been suppressed by the stable configuration of the maturity phase move in. This is a time of innovation and restructuring, laying the groundwork for a return to another growth phase.
The behavior of managed ecosystems is shaped by three properties: the accumulation of potential, the degree of connectedness between elements, and the resilience of the system in the face of shocks. The same properties apply to complex human enterprises like modern communications.
The adaptive cycle alternates periods of gradual accumulation of potential (e.g. biomass, socio-economic capital or know-how, depending on the kind of system) with sudden and often unexpected disruptions that reorganize that potential. Connectedness is high at maturity, but that is also the time when resilience to shocks is at its lowest. This cycle of aggregation followed by restructuring leads to innovation; but the release phase is often a surprise, and frequently an unpleasant one for those who were successful in the maturity phase. It is thus often experienced as a crisis.
Decision Environments
One can recognize the phases of the adaptive cycle in the internet/web, and in the larger system of communications governance. It is helpful to parse the system into four decision environments that represent different hierarchical layers:
- Political system: local, state and federal politicians seeking to advance their causes
- Inter-organizational system: peer agencies with partially overlapping responsibilities, such as the FCC, FTC and NTIA
- Organizational system: an agency, in our case the FCC, acting on its “subject” layer, and other organizations, in a context provided by the political systems
- Market/culture system: companies and citizen/consumers using technology (goods and services) to achieve their various ends, often at odds with each other and other levels of system
- Political: The political system went through a release phase with the 2008 election, and will spend 2009 in reorganization as players who have been out of office for eight years move into newly opened positions of power (cf. ecological niches), bringing new perspectives with them.
- Inter-organizational: The new Administration will bring necessarily bring changes at the top of the FTC and NTIA as well, but the consequences may not be as dramatic as those at the FCC, providing some stability at this layer
- Organizational: The FCC is due for “release” with the appointment of new Commissioners and Chairman in 2009. There is anecdotal evidence that large-scale departures of long-serving career staff in the last couple of years represent a release in itself, with the breakup of long-standing networks of expertise and the dissipation of institutional knowledge.
- Market/culture: The productive parts of the communication system are in or near maturity. Traditional content industries like news, music publishing and TV at maturity, and some are entering release. Telecoms went through a re-organization following the Telecoms Act of 1996, and is in a growth stage, judging by the consolidation of AT&T and Verizon. Similarly, the disruptive market-oriented allocation of spectrum through auctions has been absorbed, and there are signs of maturity in the concentration of spectrum in a few hands. There are still pockets of reorganization left over from the last cycle, e.g. cable taking voice share from wire line telcos, and telcos threatening cable’s video business. For all the hype, the PC/internet/web subsystem is well along in the growth phase and nearing maturity (e.g. Microsoft, Cisco, Google). Consumer habits have adapted to internet and the web, and have become mature.
Another hallmark of complex adaptive systems – and one of the hardest challenges for a regulator – is unexpected novelty. Changes in the state of a complex system are usually unexpected, in part because many dynamics are invisible. Surprises are particularly noticeable when they lead to release.
Here are some recent reminders that the innovation that we expect from complex systems usually comes as a surprise:
- Digital satellite radio expected to compete with traditional radio, not to be swamped by the iPod
- Digital video as an alternative to broadcast TV came to prominence as low-quality, user-originated content on YouTube, rather than as high quality Video on Demand via cable or DSL
- The explosion of Wi-Fi (and CDMA cellular telephony) was the consequence of esoteric decisions about unlicensed rules by the FCC in the mid 1980’s
- The collapse of music publishing – the industry lost a third of its revenues between 1999 and 2006
- The eclipse of commercial encyclopedias by user-produced content on Wikipedia
Many surprises come from contagion between problem domains that were previously considered distinct. XM/Sirius’s problems came at the intersection of personal computing devices with broadcasting; music publishing’s crisis arose from software and networking innovations that led to the P2P distribution of digital content; and the open source software movement informed Wikipedia.
Problem scope
A consequence of interlocking decision environments and intersecting problem domains is that the unit of analysis for the FCC is no longer a distinct, largely independent, well-defined industry associated with a particular technology and its own Title in the Communications Act.
Attention needs to shift from industries to problem domains, and away from solutions for a particular industry, technology and even institution or statute. For example, a policy imperative like lawful intercept is no longer limited to telephony, which leads to conflicts such as the competing definitions of information services in CALEA and the Communications Act. This is an example of the importance of the Big Picture principle for managing adaptive system. (I’ll review this principle and its three companions – Delegation, Flexibility and Diversity – in the next post.)
However, simply broadening some existing statute to cover all new possibilities is counter-productive. It conflicts with the other three principles, and falls victim to the fallacy that narrow-minded control of a single variable leads to a healthy outcome; in adaptive systems, it leads eventually to an even worse crisis.
In conclusion, the FCC is really facing a system problem, not an institutional one. Even if today’s procedural problems within the Commission were completely solved, it would not address the challenges of a qualitatively more complex and unpredictable regulation “subject”, that is, the market/culture system where innovation and growth takes place. Nor would it speak to the problems faced at the political level where the social acceleration of time poses existential challenges to the rule of law, and profoundly complicates the separation of powers between the legislature, executive, and judiciary market capitalism, and liberal democracy.
I’ll turn to the question of how the FCC should respond in the next post.
Further reading
The adaptive cycle:
Holling, C S, Lance H Gunderson and Donald Ludwig, “In Quest of a Theory of Adaptive Change”, Ch 1 of Gunderson, Lance H and C S Holling, Panarchy: Understanding transformations in human and natural systems, Island Press (2002). PDFDecision environments and the challenges individuals face in managing adaptive systems:
Ten Conclusions from the Resilience Project
Westley, Frances, “The Devil in the Dynamics: Adaptive Management on the Front Line”, Ch. 13 in Gunderson, Lance H and C S Holling, Panarchy: Understanding transformations in human and natural systems, Island Press (2002)A discussion of the intersection between system resilience, the rule of law, and Scheuerman’s notion of the social acceleration time
Cherry, Barbara A (2008), “Institutional Governance for Essential Industries Under Complexity: Providing Resilience Within the Rule of Law” CommLaw Conspectus (forthcoming)An account of the early history of civil spread spectrum
Early Civil Spread Spectrum History, Mike Marcus web siteCollapse of the music industry
theweek.comGrowth of cable voice traffic
economist.com
redorbit.com
gigaom.com
Tuesday, December 02, 2008
Reforming the FCC: Diagnosis
The financial crisis has called into question how markets are regulated; calls for reforming the FCC have been growing louder for some years. The legal/regulatory shortcomings of the FCC are a topic of frequent conversation (e.g., GMU Sep 08, PK/Silicon Flatirons Jan 09. It is therefore instructive to ask why it has ended up in this situation. Some of the problems are due to the personalities and politics of the moment, and are thus transitory. Some are due to its terms of operation; the FCC’s structure and mission are determined by the Communications Act, and won’t change fundamentally unless the Act changes. The deeper cause, which most interests me, is a change in the nature what is being regulated: the transformation of the communications business from telecoms+broadcasting to the internet.
Since the mid-90s, the computer, information and communication services have come to dwarf telecommunications services. For example, the graphic at the top of this post charts the service exports of the OECD countries according to the OECD Communications Outlook 2007 (p. 256). This was not only a quantitative change; computing brought a qualitative change. The internet/web is modular, decentralized, self-organizing, adaptive and diverse on a fundamentally different scale to telecommunications (Internet Governance as Forestry). These are all characteristics that distinguish complex systems from merely complicated ones.
An analogy may help: the FCC in the telecoms era was like a farmer managing agricultural production; today it is like a ranger responsible for a wilderness. A farmer can decide which crops to cultivate, where to plant them, and when to rotate – though the plants do the work of converting sunlight to carbohydrate, and the animals convert food to meat. Some inputs, like weather and market conditions, are unpredictable, but many – irrigation, fertilizer, seed type, antibiotics – are under the farmer’s control. (And even weather and market risk is mitigated by massive government subsidies for major crops.) The desired output is well-defined and easily measurable. Rangers, on the other hand, have to deal with a very different balance of power and responsibility. They have to protect endangered species, prevent catastrophic fires, and provide access to citizens, but have little or no control over the animals and plants in the ecosystem, or the inputs in the form of weather, migrating animals, or pests.
This limited control implies that detailed, rule-based regulation is no longer sustainable. An approach based on principles, supported by tools such as transparency and computer simulation, is the only viable strategy. Rules can determine which crop hybrid to use for a particular market need given climate and soil type; but principles – such as flexibility, taking a big picture view, fostering diversity, and delegating responsibility – are unavoidable when managing an ecosystem.
In a New Yorker article about the financial crisis, James Surowiecki uses a sport analogy to explain the difference between principles and rules:
It’s something like the difference between football and soccer. Football, like most American sports, is heavily rule-bound. There’s an elaborate rulebook that sharply limits what players can and can’t do (down to where they have to stand on the field), and its dictates are followed with great care. Soccer is a more principles-based game. There are fewer rules, and the referee is given far more authority than officials in most American sports to interpret them and to shape game play and outcomes. For instance, a soccer referee keeps the game time, and at game’s end has the discretion to add as many or as few minutes of extra time as he deems necessary. There’s also less obsession with precision—players making a free kick or throw-in don’t have to pinpoint exactly where it should be taken from. As long as it’s in the general vicinity of the right spot, it’s O.K.Pursuing this metaphor, the FCC is not only the referee of a football game, it also makes the rules – often as the game goes along.
--- James Surowiecki, Parsing Paulson, The New Yorker, 2 Dec 2008
I’ll suggest some possible ways for a new FCC to manage the new communications business in an upcoming post. However, a caveat: The ICT business hasn’t had a crisis of melt-down proportions, as finance has had, to concentrate the mind. It remains to be seen how the change in power in DC will affect this process. Some of the loudest calls for change at the FCC have come from the Right, arguing that the FCC regulates too much and too intrusively; the Left has chimed in, arguing that it regulates too ineffectively. With Democrats now in control of both Congress and the Administration, and the GOP in some disarray, the pressure to reform the FCC may well abate; calls for its abolition will certainly have less resonance.
Friday, November 21, 2008
More on Intelligibility vs. Transparency
A commentary by Richard Thaler and Cass Sunstein, the co-authors of Nudge, also notes that the growing complexity of the financial world needs more attention; cf. my recent post Lessons for communications regulation from banking complexity.
I’ve been thinking about bounded rationality for some time; see the Hard Intangibles thread. It’s one of the fundamental challenges of managing complex adaptive systems. I, like many others, recommended disclosure (aka transparency) as a key tool for internet governance; see e.g. my Internet Governance as Forestry paper).
However, the more I think about transparency, the more skeptical I become. I’ve concluded that in finance, at least, the problem isn’t disclosure but intelligibility; see e.g. my post From transparency to intelligibility in regulating finance. While it’s true that eliminating complexity could stifle innovation, that’s a false choice; nobody’s proposing to eliminate either complexity or innovation. It’s our infatuation with novelty, as well as our greed, that got us into this problem, and we have to manage our urges in both respects.
I suspect that one can make the intelligibility argument just as well for computing & communications as for finance – though the lack of a Comms Chernobyl will make it harder to sell the idea in that industry.
Transparency/intelligibility need not be mandatory; companies should be able to choose obscurity. However, the choice of keeping their activities shrouded will bear the cost of increased regulatory scrutiny – and perhaps higher expectations regarding performance against public interest mandates. Comcast need not explain exactly how it manages its network; but if it chooses obscurity, it should face tougher network neutrality expectations. Microsoft does not have to disclose its interfaces; but if it chooses obscurity, it should face a tougher anti-trust test. Google need not explain how it uses DoubleClick data to improve ad click-throughs; but if it chooses obscurity, it should face tougher privacy protection requirements.
My “transparency to intelligibility” post proposed algorithmic complexity as a way to measure intelligibility. That’s not the only method. Another (prompted by the financial crisis, and teetering software stacks) is an abstraction ladder: the more steps between a derivative and its underlying asset, the higher it is on the abstraction ladder, and the less intelligible and more risky it should be deemed to be. In computing & communications as in finance, the abstraction ladder counts the number of rungs up from atoms. The networking stack is an example: from wires in the ground one climbs up to links, networks, sessions, applications. On the premise that atoms are easier to observe than bits, and that piling up inscrutable and unstable combinations are easier the higher you go, services at higher layers will be subject to closer regulatory scrutiny, other things (like market concentration) being equal.
Not so long ago, most mortgages were of the 30-year fixed-rate variety. Shopping was simple: find the lowest monthly payment. Now they come in countless forms. Even experts have trouble comparing them and a low initial monthly payment can be a misleading guide to total costs (and risks). A main cause of the mortgage crisis is that borrowers did not understand the terms of their loans. Even those who tried to read the fine print felt their eyes glazing over, especially after their mortgage broker assured them that they had an amazing deal.Thaler & Sunstein conclude that regulators therefore need to help people manage complexity and resist temptation. They reject the option of requiring simplicity, on the grounds that this would stifle innovation, and they recommend that disclosure is improved.
Yet growing complexity on the borrowers’ side was trivial compared with what was going on at the banks. Mortgages used to be held by the banks that initiated the loans. Now they are sliced into mortgage-backed securities, which include arcane derivative products.
--- Human frailty caused this crisis, Financial Times, 11 November 2008. Thanks to Andrew Sterling for the link.
I’ve been thinking about bounded rationality for some time; see the Hard Intangibles thread. It’s one of the fundamental challenges of managing complex adaptive systems. I, like many others, recommended disclosure (aka transparency) as a key tool for internet governance; see e.g. my Internet Governance as Forestry paper).
However, the more I think about transparency, the more skeptical I become. I’ve concluded that in finance, at least, the problem isn’t disclosure but intelligibility; see e.g. my post From transparency to intelligibility in regulating finance. While it’s true that eliminating complexity could stifle innovation, that’s a false choice; nobody’s proposing to eliminate either complexity or innovation. It’s our infatuation with novelty, as well as our greed, that got us into this problem, and we have to manage our urges in both respects.
I suspect that one can make the intelligibility argument just as well for computing & communications as for finance – though the lack of a Comms Chernobyl will make it harder to sell the idea in that industry.
Transparency/intelligibility need not be mandatory; companies should be able to choose obscurity. However, the choice of keeping their activities shrouded will bear the cost of increased regulatory scrutiny – and perhaps higher expectations regarding performance against public interest mandates. Comcast need not explain exactly how it manages its network; but if it chooses obscurity, it should face tougher network neutrality expectations. Microsoft does not have to disclose its interfaces; but if it chooses obscurity, it should face a tougher anti-trust test. Google need not explain how it uses DoubleClick data to improve ad click-throughs; but if it chooses obscurity, it should face tougher privacy protection requirements.
My “transparency to intelligibility” post proposed algorithmic complexity as a way to measure intelligibility. That’s not the only method. Another (prompted by the financial crisis, and teetering software stacks) is an abstraction ladder: the more steps between a derivative and its underlying asset, the higher it is on the abstraction ladder, and the less intelligible and more risky it should be deemed to be. In computing & communications as in finance, the abstraction ladder counts the number of rungs up from atoms. The networking stack is an example: from wires in the ground one climbs up to links, networks, sessions, applications. On the premise that atoms are easier to observe than bits, and that piling up inscrutable and unstable combinations are easier the higher you go, services at higher layers will be subject to closer regulatory scrutiny, other things (like market concentration) being equal.
Monday, November 17, 2008
Lessons for communications regulation from banking complexity
A New Scientist story on Why the financial system is like an ecosystem (Debora Mackenzie, 22 October 2008) traces how the science of complexity might prevent future breakdowns of the world’s financial system.
The lessons apply to communications regulation, too. Both finance and the ICT business (“Information & Computer Technology”) are complex systems. The recommendations in the article resonate with the conclusions I came to in my paper Internet Governance as Forestry. This post explores some of the resonances.
New Scientist observes:
The question for communications regulation is whether phase changes such as those we’ve seen in finance and ecosystems have occurred, or could occur in the future. Other than the periodic consolidation and break-up of telecom monopolies, and the vertical integration of the cable and media businesses in the 80s, conclusive evidence of big phase transitions in communications is hard to find. Is there currently a slow accumulation of small changes which will lead to a big shift? There are two obvious candidates: the erosion of network neutrality, and growth of personal information bases (cf. behavioral advertising, Phorm, more).
The New Scientist article suggests that unremarked linkages, such as the increase in cross-border investments since 1995, allowed the collapse of the US real estate market to reverberate around the world. The most obvious linkage in communications is “convergence”, the use of the same underlying technology to provide a myriad of services. Common technology facilitates commercial consolidation in infrastructure equipment (e.g. Cisco routers), tools (e.g. Microsoft’s web browser), and services (e.g. Google advertising). Convergence ties together areas of regulation that used to be distinct. For example, TV programs are distributed through broadcasting, cable, podcasts, mobile phones; how should one ensure access to the disabled in this situation? There are also links from one network layer to another, as internet pipe providers use Phorm-like technologies to track which web sites their users visit.
Increased connectivity makes the financial system less diverse and more vulnerable to dramatics shifts. “The source of the current problems is ignoring interdependence," according to Yaneer Bar-Yam, head of the New England Complex Systems Institute in Cambridge, Massachusetts. Telecoms convergence creates a similar risk, with substantial horizontal concentration: Cisco has 60% market share in core routers, Internet Explorer holds 70% web browser share, and Google has 60% search share and 70% online advertising share. While modularity and thus substitutability of parts in the internet/web may limit this concentration, it needs to be carefully monitored, as captured by my Diversity principle: “Allow and support multiple solutions to policy problems; encourage competition and market entry.” Integration is a successful strategy (cf. Apple) that some find disconcerting (cf. Zittrain); it is likely to become more pervasive as the industry matures.
Diversity allows ecosystems to remain resilient as conditions change. In the quest to achieve these results, regulators have to be careful to avoid rigidity, a temptation because the financial system is so fluid. Here’s Bar-Yam again, from the New Scientist article: “Governments will have to be very careful, and set rules and limits for the system without actually telling people what to do.” To manage this risk in the comms context, I proposed the principles of Delegation (most problems should be solved by the market and society, not by government; government's role is to provide proper incentives and guidance, and to intervene to solve critical shortcomings) and Flexibility (determine ends, not means; describe and justify the outcomes sought, not the methods to be used to achieve them).
The article closes by quoting Bar-Yam: “At its core the science of complex systems is about collective behaviour.” He goes on to say that economic policy has so far failed to take into account the complexity and consequent unpredictability of such behavior, and calls for the use of testable models. This will be important in communications regulation, too. Simulations of the internet/web can help to improve policy makers’ intuition about unpredictable systems with many variables. Exploring the consequences of policy choices in simulation can identify which courses of action are most robust under a variety of possible outcomes. It’s the 21st Century version of letting states and regions experiment with regulation, which is eventually pre-empted by federal rules. Policy simulation will allow decision makers to “sweat in training rather than bleed in combat.” Since any solution embodies a set of assumptions and biases, constructing a wide range of simulations can expose hidden preconceptions. They can then eliminate policy choices that work in only a narrow set of circumstances, leading to more resilient final measures.
Update 28 Nov 2008:
I came across a very apposite comment on the value of simulation in the New Scientist editorial for the July 19, 2008 issue (No. 2665). The editorial is a critique of mainstream economics disinterest in agent-based models. It closes by saying:
The lessons apply to communications regulation, too. Both finance and the ICT business (“Information & Computer Technology”) are complex systems. The recommendations in the article resonate with the conclusions I came to in my paper Internet Governance as Forestry. This post explores some of the resonances.
New Scientist observes:
“Existing economic policies are based on the theory that the economic world is made up of a series of simple, largely separate transaction-based markets. This misses the fact that all these transactions affect each other, complexity researchers say. Instead, they see the global financial system as a network of complex interrelationships, like an electrical power grid or an ecosystem such as a pond or swamp”Consequently, the accumulation of small, slow changes can trigger a sudden crisis. Johan Rockström of the Stockholm Environment Institute is quoted as saying,
"Slow changes have been accumulating for years, such as levels of indebtedness. None on their own seemed big enough to trigger a response. But then you get a trigger - one investment bank falls - and the whole system can then flip into an alternative stable state, with different rules, such as mistrust."This is reminiscent of my Big Picture principle, which can be summarized as “take a broad view of the problem and solution space; prefer generic to sector-, technology-, or industry-specific legislation.”
The question for communications regulation is whether phase changes such as those we’ve seen in finance and ecosystems have occurred, or could occur in the future. Other than the periodic consolidation and break-up of telecom monopolies, and the vertical integration of the cable and media businesses in the 80s, conclusive evidence of big phase transitions in communications is hard to find. Is there currently a slow accumulation of small changes which will lead to a big shift? There are two obvious candidates: the erosion of network neutrality, and growth of personal information bases (cf. behavioral advertising, Phorm, more).
The New Scientist article suggests that unremarked linkages, such as the increase in cross-border investments since 1995, allowed the collapse of the US real estate market to reverberate around the world. The most obvious linkage in communications is “convergence”, the use of the same underlying technology to provide a myriad of services. Common technology facilitates commercial consolidation in infrastructure equipment (e.g. Cisco routers), tools (e.g. Microsoft’s web browser), and services (e.g. Google advertising). Convergence ties together areas of regulation that used to be distinct. For example, TV programs are distributed through broadcasting, cable, podcasts, mobile phones; how should one ensure access to the disabled in this situation? There are also links from one network layer to another, as internet pipe providers use Phorm-like technologies to track which web sites their users visit.
Increased connectivity makes the financial system less diverse and more vulnerable to dramatics shifts. “The source of the current problems is ignoring interdependence," according to Yaneer Bar-Yam, head of the New England Complex Systems Institute in Cambridge, Massachusetts. Telecoms convergence creates a similar risk, with substantial horizontal concentration: Cisco has 60% market share in core routers, Internet Explorer holds 70% web browser share, and Google has 60% search share and 70% online advertising share. While modularity and thus substitutability of parts in the internet/web may limit this concentration, it needs to be carefully monitored, as captured by my Diversity principle: “Allow and support multiple solutions to policy problems; encourage competition and market entry.” Integration is a successful strategy (cf. Apple) that some find disconcerting (cf. Zittrain); it is likely to become more pervasive as the industry matures.
Diversity allows ecosystems to remain resilient as conditions change. In the quest to achieve these results, regulators have to be careful to avoid rigidity, a temptation because the financial system is so fluid. Here’s Bar-Yam again, from the New Scientist article: “Governments will have to be very careful, and set rules and limits for the system without actually telling people what to do.” To manage this risk in the comms context, I proposed the principles of Delegation (most problems should be solved by the market and society, not by government; government's role is to provide proper incentives and guidance, and to intervene to solve critical shortcomings) and Flexibility (determine ends, not means; describe and justify the outcomes sought, not the methods to be used to achieve them).
The article closes by quoting Bar-Yam: “At its core the science of complex systems is about collective behaviour.” He goes on to say that economic policy has so far failed to take into account the complexity and consequent unpredictability of such behavior, and calls for the use of testable models. This will be important in communications regulation, too. Simulations of the internet/web can help to improve policy makers’ intuition about unpredictable systems with many variables. Exploring the consequences of policy choices in simulation can identify which courses of action are most robust under a variety of possible outcomes. It’s the 21st Century version of letting states and regions experiment with regulation, which is eventually pre-empted by federal rules. Policy simulation will allow decision makers to “sweat in training rather than bleed in combat.” Since any solution embodies a set of assumptions and biases, constructing a wide range of simulations can expose hidden preconceptions. They can then eliminate policy choices that work in only a narrow set of circumstances, leading to more resilient final measures.
Update 28 Nov 2008:
I came across a very apposite comment on the value of simulation in the New Scientist editorial for the July 19, 2008 issue (No. 2665). The editorial is a critique of mainstream economics disinterest in agent-based models. It closes by saying:
“Although the present crisis was not caused by poor economic models, those models have extended its reach by nurturing the complacent view that markets are inherently stable. And while no one should expect better models alone to prevent future crises, they may give regulators better ways to assess market dynamics, detect early signs of trouble and police markets.”
Tuesday, October 21, 2008
From transparency to intelligibility in regulating finance
As I've been reading coverage of the rolling financial crisis, it occurred to me again that one of the tools that I’ve recommended for managing complex regulatory systems - transparency - might be relevant, though with a tweak.
More than lack of transparency, a root cause of the melt-down seems to have been a lack of intelligibility. The constant refrain is that nobody understood the ramifications of the financial instruments they were creating, managing or (not) regulating.
I’ve explored this topic in the context of software development on this blog (the hard intangibles thread), and I'm now convinced that the cognitive constraints that lead to problems on large software projects apply in finance, too. As I blogged last August:
The current approach is that complex novel approaches are left unregulated, on the assumption that only “informed investors”, those who are supposedly smart enough to understand the risks, will be exposed to losses. We’ve learned that this is not the case: the informed investors are pretty dumb, and the rest of us pay for their ignorance.
I now think we should invert the regulatory presumption: the more complicated an instrument, the more firmly it should be supervised.
The hard question is how to measure the intelligibility of financial instruments in order to decide if they deserve additional scrutiny. The Mom Test for user interface design - "would your mom be able to figure this out?" – seems reasonable, but it’s hard to see how the SEC would use it in practice. A more commonly used equivalent, the Politician Test, doesn’t help either since the comprehension of politicians is a function of campaign contributions.
We’re left with algorithmic complexity: the length of the program required to specify the object. Financial wizards will surely plead commercial confidentiality in order to avoid disclosing their algorithms; but a private assessment by an impartial regulator need not lead to a leakage of competitive advantage.
More than lack of transparency, a root cause of the melt-down seems to have been a lack of intelligibility. The constant refrain is that nobody understood the ramifications of the financial instruments they were creating, managing or (not) regulating.
I’ve explored this topic in the context of software development on this blog (the hard intangibles thread), and I'm now convinced that the cognitive constraints that lead to problems on large software projects apply in finance, too. As I blogged last August:
“The sub-prime mortgage debacle is a problem of cognitive complexity. A lack of understanding of the risks entailed by deeply nested loan relationships is leading to a lack of trust in the markets, and this uncertainty is leading to a sell-off.”At that point I hesitated to draw the corollary: that limits should be imposed on the complexity of the intangible structures we create.
The current approach is that complex novel approaches are left unregulated, on the assumption that only “informed investors”, those who are supposedly smart enough to understand the risks, will be exposed to losses. We’ve learned that this is not the case: the informed investors are pretty dumb, and the rest of us pay for their ignorance.
I now think we should invert the regulatory presumption: the more complicated an instrument, the more firmly it should be supervised.
The hard question is how to measure the intelligibility of financial instruments in order to decide if they deserve additional scrutiny. The Mom Test for user interface design - "would your mom be able to figure this out?" – seems reasonable, but it’s hard to see how the SEC would use it in practice. A more commonly used equivalent, the Politician Test, doesn’t help either since the comprehension of politicians is a function of campaign contributions.
We’re left with algorithmic complexity: the length of the program required to specify the object. Financial wizards will surely plead commercial confidentiality in order to avoid disclosing their algorithms; but a private assessment by an impartial regulator need not lead to a leakage of competitive advantage.
Subscribe to:
Posts (Atom)