Fingercerting: an alternative to DRM or collective licensing

There was good news for Audible Magic yesterday when MySpace announced that it would use their software to filter out uploads that infringed copyright. Recognizing media clips using fingerprinting (more) has become fashionable as content owners begin to sue hosters.

Fingerprinting, combined with digital certificates, offers a way around the drawbacks of two currently favored ways to govern digital media use. I will focus here on video, since I recently attended a workshop on the future of video copyright at the USC Annenberg Center.

The core of the digital copyright problem is reconciling two valid interests:

Interest 1: Creators’ need to be compensated in order to cover costs and encourage more creation.

Interest 2: Consumers’ ability to make copies of copyrighted material under limited circumstances (loosely, “fair use”).

Here are two fashionable approaches to solving this problem. Each is biased to addressing one of these interests, while ignoring the other.

Solution 1: DRM

In this model, content is locked by DRM under terms specified by the creator/distributor. The consumer can only get access by observing these rules; circumvention is prevented (in the US) by the reverse-engineering terms of the DMCA.

A major difficulty arises in the intersection of DRM with fair use, since the criteria for fair use cannot be encoded in machine-executable form. Thus, Interest 2 above is generally not respected. Other difficulties include the vulnerability to a single hack that puts a piece of content in the clear, particularly if hosted off-shore; the anti-trust consequences of Content/CE/IT standardization; and usability problems with consumer experience.

Solution 2: Collective Licensing

In this model, ISPs would pay a monthly license fee on behalf of each subscriber. This would then be distributed among rights holders a la BMI/ASCAP (cf. EFF’s proposal for music).

A difficulty arises because content creators lose the ability to negotiate their compensation with consumers, thus undermining Interest 1. Owners would be compensated on the basis of some rigid formula determined by the collecting agency. Other difficulties include deriving a formula, since video isn’t as homogeneous as music; anti-trust issues in a collection monopoly; and charging users on enterprise rather than consumer networks. Option 2 also implicitly assumes that DRM is outlawed; if it were allowed to remain, then content creators could get two bites of the apple.

Another way: Fingerprinting + Certificates = FingerCerting

Option 1, the DRM approach, puts the control of content on the user’s device; however, the control is draconian and makes accepted uses like sharing around a user’s personal domain or fair use clumsy at best. Option 2, collective licensing, removes content control by levying a blanket license fee on all broadband subscribers through their ISP, but at the cost of creating an inflexible collecting monopoly and outlawing DRM.

In the “FingerCert” approach, fingerprinting is used to identify content, and an accompanying digital certificate (or “cert”) indicates that the owner has approved its transmission. If the content is registered as copyrighted but not accompanied by a valid digital certificate, an intermediary (ISP or hoster) is obliged to block it. There can still be a negotiation between an owner and a purchaser, but DRM isn’t required, only attaching a cert to indicate a contract. Once the media has been delivered, the cert can evaporate. If the media is provided without encryption, the end user can make copies for fair use without having to worry about arcane and unexpected restrictions.

The big problem with digital media is not personal copies; it’s large-scale illegal distribution. Content owners could use light-weight DRM as a “bump in the road” to mark their rights, but heavyweight (and futile) restrictions intended to prevent even a single hack won’t be necessary. This means a good experience for the vast majority of users who are happy to pay for content, but who would be deterred from buying if DRM were rigorous enough to persuade content executives that their assets were protected against all possible infringement. If you’re only willing to sell sandwiches wrapped in bank vaults, you won’t sell many sandwiches. FingerCerting prevents large-scale distribution by stopping the flow across the Internet, not in someone’s house or between friends’ iPods; it addresses thepiratebay.org and AllofMP3.com, not somebody making a mash-up for their friends. The gates don’t have to be in many places – just the major intersections, like big content sites, or perhaps just at the major IXCs.

FingerCerts gives content owners a way to control distribution of their content (protecting Interest 1), while allowing them to do so without harsh DRM that undermines fair use copying (protecting Interest 2).

What Fingercerting Isn’t

FingerCerting doesn’t require watermarking, that is, embedding (often hiding) a copyright notice in a file. Fingerprinting sets out to recognize the file from its visible characteristics. Watermarking, just like fingerprinting, has to be keep working even when videos are manipulated, e.g. by cropping or transcoding. My uneducated guess is that fingerprinting is more robust in these cases than watermarking since it’s not trying to hide the indicia.

FingerCerting doesn’t require DRM, but neither does it preclude it. It creates an environment where DRM isn’t essential to protecting mass abuse of copyright, and hopefully takes the sting out of the argument over this technology.

Challenges

Any solution to a complex problem will have weaknesses. Here are some I can think of regarding FingerCerting:

Will you need a standard for fingerprints? Audible Magic has a mechanism to register media and recognize clips; so do other companies like Philips. Cert standards exist, but one can imagine different content owners using different solutions. The complexity may be too great for intermediaries if they have to support more than a small number of mechanisms.

Packet inspection technologies to do stream identification are available. Attaching certs to streams is a different issue; I can imagine solutions, but I haven’t stumbled across any yet. Pointers, please.

False negatives – not recognizing an illegal file or stream – will occur, but that’s OK; large scale distribution can stopped since intermediaries will have multiple shots at catching streams. The bigger problem is false positives, that is, when an intermediary mistakenly blocks content. This will annoy users, and present a wonderful scenario for denial of service attacks.

Content hosters/routers will have to be motivated, by litigation or legislation, to implement such a scheme. Current US law provides a disincentive to implementing fingerprinting: Google/YouTube would rather not know that it’s hosting infringing content, because that increases its liability under the DMCA. I presume some legislation or regulation would be required to set up the incentives for a fingercerting process; I don’t know if it will be more or less onerous than that required for DRM (cf. the DMCA) or for collective licensing.

Mus Gnarus

I made a big deal about the limits to our knowledge of what we don’t know in Incognita Incognita. Reality check: even rats understand the limits of their knowledge, so I shouldn’t get too carried away.

In The Rodent Who Knew Too Much in ScienceNOW (8 Mar 2007, subscription required), Gisela Telis reports on a study that tested the self-knowledge of rats. The experimenters trained rats to understand that they could get a big food reward (six pellets) if they correctly distinguished a long sound from a short one. They could boycott the test if they wanted to, though, and go for a smaller but guaranteed reward (three pellets). As the sounds became harder to distinguish, the rats would opt out and go for the certain, though less generous, reward.

The ability to gauge one’s own knowledge is known as metacognition. We know that humans can do it, and it’s been demonstrated in monkeys and dolphins; this is the first time the effect has been shown in smaller-brained animals.

Metacognition, or “thinking about thinking,” is a strategic activity. It allows us to reflect on a (tactical) cognitive lack, such as having a blind spot for taking immediate steps to remember the name of someone you’re introduced to. (“A pleasure to meet you, John. So tell me, John, did you enjoy the lecture? You know, I agree with that assessment, John.”) Metacognitive strategies can be learned – which gives me hope that any conclusions I might draw about Hard Intangibles will lead to more effective thinking.

P.S. Latin doesn’t seem to distinguish between "rat" and "mouse" (mus). Gnarus means "knowing" or "expert."

Defining the Internet

A discussion started on slashdot last night about a succinct layman’s definition of the Internet.

Most of the examples were technical, and related to computers connected in some way. There were some metaphors: highways, trains, telephones, mail. There were a few references its social aspects. Anthropomorphism was pervasive: computers talking to each other, sending messages, sharing information with each other.

Most of the discussion was about what it was, with some comments about how it works, and occasional references to its social function.

Here’s a précis of the definitions given so far:

• collection of ideas
• bunch of connected computers
• information as trains running on tracks
• general purpose communication system
• means for computers to connect to each other and share information
• everybody already knows what it is
• computers talking to other computers over cables
• roadway, highway
• telephone system with computers calling computers
• global public computer network
• mail system
• physical: computers sending messages; social: virtual community; functional: way to use computers to send messages; technical: computers using protocols
• agreement (protocol) about how to have networks talk to each other

The best paper I’ve seen on this topic is Susan Crawford’s “Internet Think,” which contrasts the very different ways that “Engineers,” “Telcos,” and “Netheads” define the Internet. Most of the slashdot discussion would fall in the Engineers category.

The metaphors used for the Internet are so stable they’re stale . . . Perhaps the clean slate movement will stir up our thinking. How about the Internet as a brain (back to the Fifties!), or an ecosystem, or a society? The asymmetry of conceptual metaphors is perceptible in the last one: it’s more common to think about society using the Internet as a model (cf. Manuel Castells) than to model the Internet by thinking about society.

Ducking hard questions: Objective vs. Subjective

In the conclusions to his 1974 paper “Structured Programming with go to Statements,” (Computing Surveys, Vol. 6, No. 4) Donald Knuth observes:

One thing we haven’t spelled out clearly, however, is what makes some go to’s bad and others acceptable. The reason is that we’ve really been directing our attention to the wrong issue, to the objective question of go to elimination instead of the important subjective question of program structure. In the words of John Brown [Knuth citation: “In memoriam . . . .”, unpublished note, January 1974], “The act of focusing our mightiest intellectual resources on the elusive goal of go to-less programs has helped us get our minds off all those really tough and possibly unresolvable problems and issues with which today’s professional programmer would other have to grapple.”

This is a useful and concrete reminder that a fixating on objective, answerable questions can miss the point. There is a certain delight in framing an objective question: it’s elegant, precise, and one can tell when it’s been answered. Some truly important questions, though, don’t lend themselves to objective formulations. This may be because they pertain to complex concepts which have so many interlocking variables that they be reduced to an intelligible logical form, and/or because they refer to notions that are ambiguous or contested. (I suspect that these two conditions, non-linear complexity and ambiguity, are related through our inability to fit the whole of a big question into a single brain.)

Incognita Incognita

It’s hard to think about what we don’t know. If we don’t know something, there’s no “thing” for our consciousness to attend to. One can imagine the unknown as the inverse of what one does know, but that’s just the known combined with the “not” operator, rather than the unknown itself. Most commonly, we tame the unknown with a name. The old mapmakers marked mysterious places as terra incognita, today’s cosmologists explain unexpected galactic dynamics by invoking dark matter, and the religious use the word God.

And of course there’s Donald Rumsfeld, he of the unknown unknown. I’m thinking here of a third category beyond his “known unknown” and “unknown unknown”: the unknowable unknown.

Even though it’s easy enough to think about not knowing, as I’m doing now, it’s not something I do very often. I seldom look at the wall of a lecture theater and realize that I don’t know what’s behind it. My thinking stops at the wall, and bounces back into the room that I can perceive.

Dogs are largely oblivious to human conversation. They don’t follow the to and fro of conversation. They are aware of the sound and some if its import, but they don’t know its meaning. In a sense, it doesn’t exist for them. In the same way, I’m ignorant of much going on inside me and around me, and I’m ignorant of the fact that I’m ignorant.

Things I know sometimes feel like places. As I learn more about a subject, I can begin to assemble the rooms representing topics into a building. But if I don’t know something (statistics, say) it’s not as if it’s the unexplored south wing of a mansion. There is no south wing. I have no sense of its shape. Something once known but now forgotten (like Green functions, in my case) are ghostly ruins remembered from a dream; there are only wisps and fragments.

We make up stuff to hide the fact that we don’t know. Helen Phillips describes in New Scientist (“Mind fiction: Why your brain tells tall tales,” 7 October 2006) how people make up stories when the reasons for their action are not available to conscious introspection:

[Timothy Wilson and Richard Nisbett] laid out a display of four identical items of clothing and asked people to pick which they thought was the best quality. It is known that people tend to subconsciously prefer the rightmost object in a sequence if given no other choice criteria, and sure enough about four out of five participants did favour the garment on the right. Yet when asked why they made the choice they did, nobody gave position as a reason. It was always about the fineness of the weave, richer colour or superior texture. This suggests that while we may make our decisions subconsciously, we rationalise them in our consciousness, and the way we do so may be pure fiction, or confabulation.

Note that people didn’t say, “I don’t know.” This is an important result for the study of hard intangibles. We are usually not aware that we have a limitation. Sometimes cannot even believe that we’re limited. Here’s another excerpt from the New Scientist story:

It is surprisingly common for stroke patients with paralysed limbs or even blindness to deny they have anything wrong with them, even if only for a couple of days after the event. They often make up elaborate tales to explain away their problems. One of Hirstein's patients, for example, had a paralysed arm, but believed it was normal, telling him that the dead arm lying in the bed beside her was not in fact her own. When he pointed out her wedding ring, she said with horror that someone had taken it. When asked to prove her arm was fine, by moving it, she made up an excuse about her arthritis being painful. It seems amazing that she could believe such an impossible story. Yet when Vilayanur Ramachandran of the University of California, San Diego, offered cash to patients with this kind of delusion, promising higher rewards for tasks they couldn't possibly do - such as clapping or changing a light bulb - and lower rewards for tasks they ould, they would always attempt the high pay-off task, as if they genuinely had no idea they would fail.

If we can observe the limitation in others, we can at least study it, if not experience it ourselves. However, it will be hard to teach others – and ourselves – to behave differently if, in our bones, we still cannot conceive of our lack.

Let’s hope we’re not rational about climate change

Global warming is a classic collective action dilemma.

A solution to global warming is a collective good and will be undersupplied, as Mancur Olson pointed out back in 1965.

Therefore, if Olson’s premises and argument are valid, we’re dooooooomed.

However: his argument supposed a rational economic agent who will wait for others to act, since his contribution is so small that on its own it won’t make a difference, and it’s absence won’t be noticed.

Only if humans don’t act as selfish rational agents will we avoid a climate catastrophe.

Fortunately, behavioral economics etc. suggests that we have bounded rationality, and even better, psychology and evolutionary biology suggests that non-rational altruism is hard wired.

Maybe there’s hope.

Algorithmic trading changes markets (maybe)

Kyril Faenov alerted me that electronic exchanges feeds back on themselves in unprecedented ways due to automated (or algorithmic) trading. Traders observe the market, and imagine a way to make money; their quants then write software to execute this trading strategy automatically. Running this code creates new, fast and extensive linkages between market processes.

Robin Sharpe provides an excellent introduction to algorithmic trading in Automated Trading and the New Markets. For more information, see John Bates in Dr. Dobbs, and wikipedia.

For example, trader A (or their software) notices a periodic spike in the price of equity X; trader B may be trying to buy a very large position of X in small portions so as not to push up the price too much. Trader A (or the software) buys stock X just moments before each predicted spike, selling it to trader B at a higher price once B enters the market. Trader B (or their software) notices the run-up, and changes their buying rhythm to disrupt trader A. And on it goes. . .

Such market interactions aren’t new, but software can execute the trades faster than humans can respond to them. Rather than duels between traders in real time, it becomes a duel between traders’ models of how the market functions. These are contests between world theories, where the theories themselves constitute the world. Kyril calls this the “reflectivity” of the market.

Douglas Hofstadter introduced the term “strange loop” in Gödel, Escher, Bach to describe a series of steps through a hierarchical system which take one back to the beginning. His new book I Am a Strange Loop uses this concept to explain self-awareness. In a New Scientist interview, Hofstadter says the brain, and the self, is like a smile because it’s a process rather than a thing. (Extending the metaphor: Software is to hardware as a smile is to a body.)

A market is observable through its behavior, that is, how it responds to stimuli. When the responses happen faster than humans can follow, and involve the integration of more variables than humans can handle on their own, the market is less a social interaction among people than an environment in which people act. A market is neither a place, nor a group of traders, nor the sequence of trades, nor a reflection of an outside commercial reality; it is the self-perpetuating process that involves all of them. The system’s behavior becomes a subconscious expression of the cumulative conscious plans of many people – subconscious because the mechanism is not directly available to human introspection.

Markets are examples of distributed cognition, that is, cognition which occurs in an ensemble of people and tools, rather than in a single brain; see Giere (2002, PDF) for a good survey. What’s striking about algorithmic trading is that the amount of cognition occurring outside human brains is growing rapidly.

Robin Sharp (ibid) points out that trading is increasingly hands-off, since humans can’t cope with the reaction times required.

“Whilst the theory behind program trading is fairly simple, the software reality is that program trading operates at a different time-scale to even the fastest human trading. . . It’s worth understanding that the brain brings experience and subjectivity to the table and software brings speed and objectivity to the table. For most types of trading experience beats speed but there is a lot of noise in the market in the sub-second region where the brain simply can’t compete with a computer. . . At the moment the volume of trading at these sub-second time scales is not be great (less than 5%) and is held back because the coarse granularity of ticks and price data has been designed for human interaction. However this will change.”

He also notes that algorithms can exploit the capacity limitations of human traders: “A large number of trades are difficult for traders to juggle in their heads. When such large events occur is small time frames computers can predict irrational behaviour of traders, again for a profit.”

The kinds of problems that traders face are not only analytical or cognitive; they’re also social. Sharp notes the organizational impediments to certain kinds of trades: “One of the reasons traders don’t trade cross market is that they cannot price the instruments fast enough, and – again – is that traditional corporate management structures and regulatory structures impede cross-market trading.”

Sharp tabulates the latencies in a trading cycle. The computer processing is 200 milliseconds, quick compared to 1,250 milliseconds for human perception, evaluation and response. The network latency is about 400 milliseconds. It’s worth noting that these network applications reinforce old geographic patterns rather than abolishing distance. Kyril tells me that the NYSE is doing a good business selling rack space at the Exchange to big trading houses; because every millisecond counts, their computers need to be on the same LAN as the Exchange or else they’ll lose to faster arbitrageurs.

I’m struggling with the question of whether automated trading leads to a qualitative change in the markets, rather than simply a quantitative one. While algorithms that respond to changes in the market begin to constitute the market, this kind of loop applies to traditional human-only trading, too.

Sure, the feedback loop is much faster with automated trading. But is it a difference in degree, or a difference in kind? It’s different from the human perspective; stuff now happens too quickly to follow consciously, and the role of humans changes. However, it might simply be a change in time scale, not a change in process.

The increasing complexity of the market may be even more important. Arbitrage can link markets, which generates more correlated variables than traders can juggle in their heads. An automated trading strategy could link current and futures markets, on different exchanges (New York and London), for different instruments (equities and foreign exchange), and different data types (Reuters news feed, GOOG and MSFT stock prices, S&P500 index, the 15 minute volume weighted price of GOOG). As Robin Sharp points out: “Eventually arbitrage will force separate markets to revaluate their relationships. It only takes one successful arbitrage engine to forever link two previously unrelated markets. Anybody in the business will know how deeply this will be felt.”

Perhaps integrating more streams of information in more complex and rapid ways creates a new kind of market causality. When humans were trading with each other, it was a social process. Now, from the human perspective, it’s more like experimenting on the world than dealing with people. “Hard intangibles” come into play because this new world is not the one humans evolved in. Humans still set the goals and strategies, but the parameters of this world interact in unexpected ways. And genetic algorithms will lead to algorithmic trades that are profoundly alien to human intuition. Trading is another activity, like large software projects, where the abstractions we’ve created are beginning to outstrip our ability to understand them.

------------

Ronald Giere, “Scientific cognition as distributed cognition,” in The Cognitive Basis of Science, Cunningham, Stich & Siegal (eds.), Cambridge University Press, 2002

Is Dampé Dead?

Richard “Dampé” Denton is (was?) the 15 year old writer of Ocarina of Time 2D, a much-anticipated Legend of Zelda title. He was supposed to have died on the 23rd March (or was it the 21st?), but Squidnews reveals why it reports of his death are a hoax. The reigning theory is that Dampé wanted to escape from the pressure of anxious fans by arranging his supposed demise.

I was struck by how easy it was for the writer to do his fact checking, searching news.com.au and Fairfax classifieds for reports of death under the name of Denton, and most interestingly, looking up the Victoria’s Transport Accident Commission road toll statistics tool, searchable by date, location, victim and injury type.

This is a great example of how findable one, or one’s lies, are is on the web, and the pressures of constant visibility.

Impossible puzzles

Dave Munger over at Cognitive Daily picked up the New York Times article on the Japanese gaming company responsible for the Sudoku craze. He recommends their Kakuro puzzles (see e.g. the Washington Post), which are rated one to four stars for difficulty. Some people are better than others at puzzles, and experience helps. But there's a limit to human cognitive capacity; as Dave says, "I've never seen a five-star puzzle."

The Times describes how Japanese puzzle solvers tinker and improve puzzles, presumably to make them just hard enough to be a challenge. If puzzles are too easy, it’s boring; if they’re too hard, there’s no point attempting them. We humans can clearly create puzzles we can’t solve ourselves. However, some of these “puzzles” aren’t games we can ignore if they’re too hard, or simplify at will. They’re critical artifacts and infrastructure like software, financial systems, and webs of trade.

Mapping E8

The AP reports that a group of mathematicians have at last characterized the E8 Lie group. I'm intrigued by the fact that it seems to be in the gray zone between the understandable and the unintelligible.

Jeffrey Adams, the project's leader and a math professor at the University of Maryland, is reported as saying, "To say what precisely it is is something even many mathematicians can't understand." The scale of E8 boggles the mind: all the information about E8 and its representations is 60 gigabytes in size.

For more on E8, see http://aimath.org/E8/. Curiously, the representation on that page is a semi-lattice - cf. my post on Christopher Alexander's application of this mathematical concept to the complexity of cities.

The language used to describe this topic is strikingly physical: "The classical groups A1, A2, A3, ... B1, B2, B3, ... C1, C2, C3, ... and D1, D2, D3, ... rise like gentle rolling hills towards the horizon. Jutting out of this mathematical landscape are the jagged peaks of the exceptional groups G2, F4, E6, E7 and, towering above them all, E8. E8 is an extraordinarily complicated group: it is the symmetries of a particular 57-dimensional object, and E8 itself is 248-dimensional!"

[Thanks to Scott Forbes for alerting me to this result.]

Pam Heath on Hard Intangibles

I'm very grateful to Pam Heath for sending me her thoughts on my recent Hard Intangibles Update. With her permission, I reproduce it here a few comments:

Pam: I really like the idea of exploring cognitive hardness and cognitive capacity. It’s fascinating. I do wonder if the challenge of parallel computing is cognitive capacity per se or something more interesting. Which is probably your point. Doh. I wonder what else/something more is going on. Certainly since we probably haven’t thought/tested/experimented enough about cognitive capacity, then we haven’t yet thought about ways to compensate for/extend it.

Pierre: I haven’t gone much beyond the cognitive capacity yet. At this point I’m trying to narrow the focus, but I’m sure you’re onto something. The “something” may be related to cognitive limitations but in a more general way, e.g. cultural constructs.

Pam: My gut says that the field of computer science is severely limited by the people who practice it, and their brains – the mindsets, the particular kind of intelligence that they have, their personality traits. Do their brains work differently, I wonder, than visual artists or doctors or lawyers or anyone else?

Pierre: Howard Gardner did very influential work decades ago about learning styles that’s relevant here. People’s brains evidently do work somewhat differently. One of the tricky issues in my project is separating individual variation from generic Home Sapiens constraints. There’s a big difference between Einstein and the village idiot, and between Einstein and Picasso, but a bigger difference between any of them and a gibbon.

Pam: Do we have different types of consciousness? Does that bring anything to bear to the problem? What is cognitive capacity, really? Is it absolute? Is it really the number of connections that we can hold in our heads at one time, or something else? Is it a spinning plate problem, or something else?

Pierre: I’m sure there are different types of consciousness, because consciousness is compound. The current scientific consensus seems to be that consciousness is constructed concurrently in many brain areas, and in fact that most of the interesting thinking we do is pre-conscious. When it comes to “capacity,” that’s just a (collection of) metrics, e.g. the number of concurrent independent variables one can handle. Back to Gardner’s multiple intelligences: he showed pretty conclusively that IQ tests just measured linguistic and mathematical facility, and that there were at least five other important skills that IQ tests didn’t track.

Pam: I also wonder what the interplay of cognitive capacity and consciousness might be. Should we think of one as a subset of the other?

Pierre: You are asking the big questions, aren’t you? Speaking from almost complete ignorance, I’d venture that they’re partially overlapping. Animals that have less cognitive capacity than we do are also conscious, but not all capacity is in consciousness.

Pam: Has computer science been limited by the models it’s used to develop software and computer science approaches and concepts? For example, would a more consilient approach make big breakthroughs and paradigm shifts? What if dev teams/architects/etc included a broader range of types of thinkers?

Pierre: More diverse teams may lead to more breakthroughs, but will also have higher coordination overheads. Part of the trouble with “wicked problems” is the social complexity engendered by multiple stakeholders who can’t agree on the problem, let alone the solution.

Pam: If “intangibility and flexibility of software presents a qualitatively different cognitive challenge to most (all?) previous kinds of engineering,” then I guess we’ll need a different kind of engineer, won’t we? Maybe we should stop thinking about it as engineering at all.

Pierre: Perhaps, but not necessarily. We may just need to train them differently, give them specific tools, and manage our expectations.

Pam: Do the brains of various language speakers work differently from each other? Do the brains of Indian, Chinese, or any other nationality of developer work differently than European or American ones? Female vs. male developers? Do the brains of deeply consilient thinkers work differently? Are some cultures more cognitively hard than others? Does each culture have its own flavor of cognitive hardness?

Pierre: I very much doubt this. Sure, there are cultural differences in math and science performance (much greater than the gender differences which are de minimis, it turns out), but my assumption is that humans don’t vary that much. That said, cultures may have found different work-arounds, and we can surely learn a lot from looking across cultures, just as we can learn (as you suggested above) by looking at people with different aptitudes.

Pam: My bias is that if we leave resolving why parallel computing is hard to those who live naturally in that world it will take longer and be less satisfactory.

Pierre: Yes indeed. Iif we can answer “why is programming hard?” we’ll also be able to cast some light on questions like “why is IPR hard?” and “why is international policy hard?”

Pam: Your proposed threads nag at me somehow. They sound logical, but incomplete. Ah, maybe because they’re all couched in terms of limitations and difficulties, and not the opposite. Trying on both approaches might give more interesting and, dare I say, valuable results.

Commons and markets

I attended a Fellows meeting last week at the Annenberg Center at USC last week. The topic that generated most interest was “the commons.” I began to suspect that “commons” is a frame that culture-studies people almost take for granted, in the same way that “markets” is the default for business-studies people. If the meeting had been organized by a business school rather than a communications center, the conversation would’ve been about markets. In both meetings, attendees would’ve argued that their world view was the most certain source of innovation.

While each side says the other is included in their approach, the terms function as shibboleths.

Commons:

1. collective, sharing, relationships, inclusion
2. abundance
3. public goods
4. cultural studies, academics
5. generates positive externalities
6. socialism
7. pro-government, state management, anti-corporation, liberal
8. open, shared
9. suspicious of profit, trusts in altruism
10. feel threatened by the market “second enclosure”, concentration of ownership
11. unlicensed spectrum

Markets:

1. competitive, exclusion
2. scarcity
3. private goods
4. economics, business
5. worry about burden of negative externalities being taxed; focus on internalities
6. capitalism
7. anti-government, pro-corporation, conservative, libertarian
8. closed, proprietary
9. trusts in profit, suspicious of altruism
10. feel threatened by loss of property rights implicit in commons rhetoric – “theft”
11. licensed spectrum

Commons and markets seem to function both as frames and as signaling devices. They’re frames because they each highlights certain aspects and suppress others; and they function as signals because someone who talks in terms of (say) commons will be trusted on a broader range of socio-political issues. Commons is a code for signaling a left-leaning political perspective; markets ditto for the right.

Conceptually they complement each other; commons and markets are like yin and yang. Each needs the other:

Markets need commons

• public goods (defense, clean air) context in which market is embedded
• common knowledge as basis for progress – incentive to publish inherent in limited time patent monopolies

Commons need markets

• farming example: raise sheep on common ground, but sell meat/wool in a market; ditto for lobster fishermen
• academics creating a knowledge commons are paid out of surplus wealth generated by market capitalism (taxes, foundations)

One can see the Internet from either perspective

• common protocols, languages
• commercialization ex VC investment: Yahoo, Google, Amazon, YouTube

That raises the question of what their superset might be. A possible containing frame for commons and markets is “decentralized coordination.” This is itself part of another dichotomy: centralized vs. decentralized coordination. An example from spectrum policy: wonks who argue about unlicensed vs. licensed allocations would agree that either is an improvement on the traditional “command-and-control” system of administration. Saussure may have been right that meaning comes from difference; in that case, there will never be a single non-contested perspective.

I’m most interested in the nexus: how do commons and markets complement each other, and how do you calculate how much of each you need? To what degree can one formalize the interdepence of markets and commons? One can do a simple calculation for real estate to show that a non-zero percentage of public parks increases property values. I’d love to do the same for spectrum, but haven’t figured out how, yet.

Hard Intangibles - Update

I’ve gained some clarity recently about what the hard intangibles project consists of. In short, I’m fascinated by the growing gap between our innate capacities and the intangible world we’re building.

I cut my teeth on the question of mental models used in spectrum policy and cognitive radio research (PDF). I’m now preparing to survey the literature for evidence that people find working with digital artifacts (computers, software, web sites, social network spaces, etc.) difficult because of the differences between the dirt world and the digital world.

The medium term goal is to answer the question “Why is (parallel) programming hard?” The dominant conversations about program hardness today are either technical (e.g. complexity classes) or social (e.g. “wicked problems”). I’m interested in a third perspective: cognitive hardness, which comes in two flavors. Bruce Schneier recently published a good paper on how cognitive biases can affect the security of computer systems. I’m more interested in cognitive capacity rather than cognitive biases, e.g. the number of independent variables we can keep in our head at the same time. These limitations are presumably at the root of folklore about the right (small) number of arguments in function calls, and the number of modules in a project. They presumably also guide programmer’s preferences for simple organizing principles like lists and trees (rather than, say, semi-lattices).

I’m not dismissing social issues; it’s clear that they are at the root of many failures in software projects. However, these problems are by no means unique to programming. I believe that the intangibility and flexibility of software presents a qualitatively different cognitive challenge to most (all?) previous kinds of engineering.

It gets even harder and more important as we turn to parallel programming. Limitations on human working memory will make grasping parallel programs particularly difficult. Programming tools are surely part of the solution, but tools have typically automated and accelerated activities that humans have already mastered. It's an open question whether we can conceive of 1000-core parallel processing sufficiently well to create tools.

The work I want to do on programming has a number of threads:

1. Review neuroscience and cog psych literature for clues about limitations that would apply to programming
2. Interview programmers to understand what they find difficult, and tie it to #1
3. Do some psych (and ideally neuro) experiments on programmers to validate hypotheses
4. Evaluate current programming methods and tools against the cognitive limitations we’ve identified to find matches and gaps
5. Identify most promising areas for improving programming by taking cognitive limitations into account

The answers on why is programming hard are key, I believe, because they will probably generalize to other hard intangibles, like law, system planning, and international relations.

Not what it seems

An LA Times story by Richard Rushfield, carried by the Seattle Times, describes fakery on the web: "[The] Web's honor code — the idea that what you are seeing is direct and real, that for every open ballot a one-user, one-vote principle will prevail — is every day being subverted." The article contains examples of the ways in which digital media are different from the dirt world, notably scale, opacity and mutability.

Rushfield tells the story of the "Bride Has Massive Hair Wig Out" viral video which, it turned out, was a promo by hair products company Sunsilk; and the story of Digg's admission that its ranking is being undermined by people trying to game the system. Deception and manipulation weren't introduced by the web, but they have been accelerated by it. The video looked hand-made, but was an ad; the scale of the Internet vaulted it onto the TV talk shows in a matter of weeks. The mutability of digital media made it impossible to tell whether it was produced professionally or by amateurs. The Digg story hinges on the site's algorithm; founder Kevin Rose has reassured us that everything is under control, and that we can trust his company's proprietary and hidden ranking algorithm.

A Program is not a City

Christopher Alexander argues in “A City is not a Tree” that cities that have grown over the years, “natural cities” like Kyoto, London, and Manhattan, are definably more complex than planned cities, like Chandigarh, Brasilia, and the British New Towns. [I found this article via Scott Rosenberg’s recent book Dreaming in Code.]

According to Alexander, planned cities are “trees,” mathematically speaking, and natural cities are “semi-lattices.” Semi-lattices contain overlapping units; “trees” and planned cities, do not. For example, the center of Paolo Soleri’s Mesa City (illustrated in the article) is divided into a university and a residential quarter, which is itself divided into a number of villages, each again subdivided further and surrounded by groups of still smaller dwelling units. Semi-lattices are more complex than trees:

“We may see just how much more complex a semi-lattice can be than a tree in the following fact: a tree based on 20 elements can contain at most 19 further subsets of the 20, while a semi-lattice based on the same 20 elements can contain more than 1,000,000 different subsets.”

Alexander’s claim is not entirely convincing – for example, he never shows that natural cities are semi-lattices – but I’m persuaded that the organic old cities we love are more complex than ones that spring fully formed from the head of an architect.

There are often disconnects between what we can recognize as good, and what we can make ourselves. Most people can appreciate skillful violin playing, but can’t do it themselves. Natural cities demonstrate that we can, collectively and over time, produce semi-lattice artifacts. (I’ve claimed that social processes are a way of dealing with problems that are to big to fit into individual brains.) Natural cities just feel right, but when even the greatest minds design a city, they resort to nested hierarchies: a tree structure. Perhaps humans can’t, through conscious intellectual effort, make a persuasive structure with a semi-lattice’s overlapping intricacy.

The cognitive capacity limitations of individual brains may mean that we can’t keep track of enough units concurrently to make generate complexity. Given the limitations on single brains, we have to use simpler rules, like trees, with less satisfactory but more controlled results. This may be why hierarchies are so deeply embedded in software:

“A city may not be a tree, as Alexander said, but nearly every computer program today really is a tree – a hierarchical structure of lines of code. You find trees everywhere in the software world – from the ubiquitous folder trees found in the left-hand pane of so many program’s [sic] user interfaces to the deep organization of file systems and databases to the very system by which developers manage the code they write.” [Scott Rosenberg, Dreaming in Code, p. 185.]

Simple rules can sometimes yield complex results, which is why fractals fascinate us. But while the Mandelbrot set might be pretty, it doesn’t speak to the heart in the way that a great city does. We can see that our minds and machines are inadequate, but we can’t as a matter of individual effort do what is required, which is (if Alexander’s right) to conjure semi-lattices. [1]

Complexity isn’t always attractive. Simplicity is appealing when it is important to understand something, that is, when we need to fit something inside our brain. Richard Stiennon made a powerful case that Windows is harder to secure than Linux by showing two pictures. The economy (and tree-like structure) of the Linux call diagram argues strongly that it is a more intelligible, and thus more easily securable, system.


-------------

[1] There seems to be a connection with the P =? NP problem which, loosely stated, asks if is always easier to demonstrate that a given solution is correct, than to find it. Many (most?) computer scientists think that P≠NP. The intuitive rightness of this unproven conjecture resonates with the fact that so often we can appreciate a thing of beauty, but can’t make it ourselves.