A view on the policy making stream

BusinessWeek writes that IBM is pushing “stream computing”, which is processing incoming information on the fly rather than putting it in a database first, and then mining it.

I’ve been trying to mine information in the FCC’s Electronic Comment Filing System (ECFS), the repository for all interactions that petitioners have with the agency. The BW story got me thinking what one could do if updates to ECFS were easily accessible on the fly, along the lines of the proposal by Ed Felten and colleagues that government should expose underlying data rather than creating portals.

One could do a lot with just the metadata, that is, cover information on who submitted a document to ECFS. A little extra processing to, say, extract information from the filed documents on all the people present in a meeting, would add a great deal of value. One could also extract information about what topics are being discussed by doing semantic analysis of comments and reports of meetings between petitioners and the agency.

Some things researchers (not to mention commercial information providers) could do with this kind of intelligence:

Track the ebb and flow of meetings related to a particular proceeding

Be notified when a specific company, company in a coalition, etc. reports a meeting with the agency, and see it in the context of other meetings by opponents and allies

Put a watch on the meetings in a particular bureau of the agency

Track the personalities – who’s meeting with whom, who hasn’t been seen lately, who seems to be a rising star. I’ve been told that John de Figuieredo predicted the importance of William Kennard before he was tapped for the FCC by noticing that he was in a lot of key lobbying meetings. (Caveat: I may have misremembered the characters in this anecdote. Please correct me if you know better...)

Given time series information one could develop leading indicators for when a proceeding was heating up, or when something big was brewing.

Of course, the Garbage In, Garbage Out Rule applies; if petitioners file late, or misrepresent their interactions (i.e. lie), all the stream computing in the world will be for naught. We may need a suggestion I heard from Bob Pepper, a former FCC staffer now at Cisco: make petitioners warrant that their submissions are true, on penalty of perjury. Curiously, there is apparently no requirement for petitioners to tell the truth, and no penalties if they lie.