EFF Launches The Great California Database Hunt


Imagine if local governments were like restaurants, where you could pick up a menu of public datasets, read the names and description, then order whatever suits your open data appetite?

This transparency advocate’s fantasy became reality in California on July 1, when a new law took effect. S.B. 272 added a section to the California Public Records Act that requires local agencies (except school districts) to publish inventories of “enterprise systems” on their websites. We are talking about catalogs of every database that holds information on the public or serves as a primary source of government data.

And we need your help on Saturday, Aug. 27 to—as the saying goes—catch ‘em all.

Similar policies are in place on the federal level due to President Obama’s 2013 Open Data Policy, which requires every federal agency to compile an inventory of its data resources and say what’s public and what’s not.

Under the new California law, these catalogs don’t just simply list the names of databases. They also contain information such as: the purpose of the system; the type of data collected; how often data is collected and updated; the name of the software product being used; and the vendor supplying it.

The passage of S.B. 272 was a victory on multiple fronts. Now, the public can look through these catalogs in order to file records requests for data sets. Privacy and civil liberties activists can also learn what kind of data is being collected on the public, including police databases and certain surveillance systems.

So far, there’s little consistency between local agencies publishing these sets. For example, the City of Manhattan Beach provides its inventory of 13 enterprise systems as a .pdf file.  Meanwhile, the City and County of San Francisco offers a robust inventory of 451 data systemsthat can be filtered, searched, sorted, and exported in multiple formats.

Currently, however, all these databases reside on individual websites.

The Electronic Frontier, the Data Foundation, and the Sunlight Foundation are now teaming up to collect links to all these data catalogs in a single repository. And we need your help.

Join us on Aug. 27 for a sprint to track down and index these catalogs across California. We’ll be holding events in San Francisco and Washington, DC, but you will also be able to join us remotely from where you are in the world.

To register for the event or for more information, just sign up. (If you plan on attending in-person in DC, please also register with the Data Foundation for logistical coordination.)