I was reading Obama administration agrees with Sunlight: Agencies should disclose what data they keep private, which is a topic I follow closely, as I'm passionate about government opening up their data inventory, but also because worked as a Presidential Innovation Fellow on helping agencies open up their data. I completely agree that agencies should have to disclose lists of even their private enterprise data repository, to help establish a clear picture of just exactly what is public or private government data.
While reading the post, I noticed the statement that, "most agencies seem to have accepted and understood the need for this change, but some concerns have been voiced":
- The Department of Commerce, which holds a very sizable amount of data, has cited “certain technological issues that are creating major barriers to creating the PDL as described [in OMB’s guidance].” This is a concern, but Commerce appears to be working through these issues with relevant officials in the General Services Administration and OMB.
- NASA has expressed significant concerns with the new process, specifically citing pushback and time constraints from its FOIA office and legal team. They claim that “identifying and indexing our open data is our priority,” and indexing its non-public data will take valuable capacity away. Fortunately, NASA has bought into the public process and are engaged in meaningful conversation about the agency's concerns on the Project Open Data Github.
It is interesting that agencies cite the technology challenges in complying with open data mandates. In my experience working on opening up government data, right after the education of the average government employee, challenges imposed by the technology they use is one of the biggest hurdles. To make my point, you can't find "Export Spreadsheet to JSON" as a Microsoft Excel option. Most systems encourage the output of PDFs from core systems, which when it is combined with lack of open data export capabilities, makes opening up of inventories extra work.
We need to continue the hard work to educate the average government employee about open data principles, and the little things they can do in their daily work, to help collectively move their agency towards being more open, and machine readable by default. However, we also need to push harder on software vendors, who are providing technological solutions to make sure the imports and exports of data in open, machine readable formats is a default feature. There also needs to be more mandatory API language in procurement contracts, requring APIs to be baked in by default in all government systems--should never be an afterthought.
There is only so much responsibility that we can place on the average government employee, trying to get their job done on a daily basis. All of the companies who are profiting off selling the government software solutions, also need to share in the responsibility, and help make sure our government is equipped with the proper tools. Government being open and machine readable by default shouldn't create extra work for agencies, it should help reduce their workload.
