Congressional Data Mining: Coming Soon?

By slipping a simple, three-sentence provision into the gargantuan spending bill passed by the House of Representatives last week, a congressman from Silicon Valley is trying to nudge Congress into the 21st Century. Rep. Mike Honda (D-Calif.) placed a measure in the bill directing Congress and its affiliated organs—including the Library of Congress and the Government Printing Office—to make its data available to the public in raw form. This will enable members of the public and watchdog groups to craft websites and databases showcasing government data that are more user-friendly than the government's own.

If the Senate passes the bill with the provision intact, citizens seeking information about Congress' activities—such as bill names and numbers, amendments, votes, and committee reports—won't have to rely on government websites, which often filter information, are incomplete, or are difficult to use. Instead, the underlying data will be available to anyone who wants to build a superior site or tool to sift through it. "The language is groundbreaking in that it supports providing unfiltered legislative information to the public," says Honda's online communications director, Rob Pierson. "Instead of silo-ing the information, and only allowing access through a limited web form, access to the raw data will make it easier for people to learn what their government is doing."

Successful, privately-created websites that provide the public with information about Congress' actions already exist. OpenCongress.org, GovTrack.us, Legistorm.com, and MAPLight.org all make legislative data available to the public in ways that are easier to navigate than Congress' primary web portal, a system called Thomas. Those sites currently get their data through techies who "scrape" Thomas and other government websites, which means they use bots to process the HTML and gather what is valuable. The process is labor-intensive and imprecise. "It's difficult to keep the data up to date, in some cases impossible, and occasionally there are errors in the data," says Josh Tauberer, the 26-year-old who runs GovTrack.us and does lots of the "scraping" that others use. "This could all be fixed by a bulk data download."

Tauberer expects that the availability of additional and easier-to-use congressional data will spur innovation. "You can expect to see other sites spring up doing new and interesting things with the information." He anticipates charts, graphs, and maps that represent congressional goings-on visually—"ways of visualizing the congressional process that we couldn't yet imagine." Honda, with his Silicon Valley roots, expects that developers and coders will quickly outpace the government's efforts to date. "We hope that we can learn from the wisdom of crowds," says Pierson.

There are government agencies that already provide massive amounts of data via databases. The Census Bureau provides huge amounts of information in raw form, allowing academics, statisticians, and think tank scholars to comb through it in any way they please. The Federal Elections Commission publishes unedited data on campaign contributions, giving rise to sites like OpenSecrets.org, which allows the public to see who is donating to whom, and allows journalists and watchdogs to investigate the influence of money in politics.

"In our Web 2.0 world, we can empower the public by providing them with raw data that they can remix and reuse in new and innovative ways," Honda told Mother Jones in a statement. (Disclosure: In the summer of 2002, I briefly worked as an intern in Honda's district office.) Honda's provision, however, pertains only to legislative data. Federal departments like the Environmental Protection Agency, the Food and Drug Administration, and the Department of Energy have reams of data that political scientists, economists, and researchers of all stripes would love to get their hands on. Many who work at the intersection of technology, politics, and transparency believe that the key player in broadening Honda's effort to include the executive branch will be Vivek Kundra, the former Chief Technology Officer of the District of the Columbia who was named Obama's Chief Information Officer on Thursday. According to the National Journal's Tech Daily Dose, Kundra "told reporters Thursday he will launch data.gov, a Web site intended to 'democratize data' by giving the public raw feeds of information from a range of agencies."

John Wonderlich, the policy director at the Sunlight Foundation, which has created or funded several tools that make government data easier to analyze, is holding out hope that the president's Open Government Directive, which is due at the end of May, will further address the issue of data availability. He applauds Honda for putting Congress, at least, on the right track. "Without Honda's attention to this issue, congressional level attention to bulk data access would be unlikely," he says. "We're happy to see this first step."