At nearly 19,000 square kilometers, Kruger National Park, which spans both South Africa and Mozambique, is one of the largest wildlife preserves in Sub Saharan Africa. It’s home to thousands of highly endangered elephants, lions, leopards, and rhinoceroses. The park is one of the last refuges with large numbers of white and black rhinoceroses. There […] […]
Image via Politwoops. This week’s roundup of deletions archived by Politwoops include a purple bedecked congresswoman, retaking a holiday from unknown detractors and a refreshing explanation on a bizarre deletion. We start with the image t… […]
Many state governments in the U.S. collect and report local government data. These data compilation efforts are often many decades old and deeply rooted in the hardcopy delivery methods that prevailed before the digital age. Recently, the California State Controller’s Office (SCO) has started to replace its traditional reports with big, open, machine-readable and visualizable data sets.
In California, state reporting on local government can be traced all the way back to a 1911 law that required cities to file financial reports with the state controller. Each year, the controller’s office compiled this data into a book. The first edition of the “Cities Annual Report” was recently digitized by Google and can be found here. In later years, the California state government began requiring counties, special districts and pension plans to submit analogous reports to SCO – also for hardcopy publication.
Over the past decade, SCO published these volumes as PDFs on its web site. As Controller’s Office spokesman Jacob Roper told me, this publication method has a number of limitations. Aside from the obvious problem of the PDF format making analysis of the underlying data difficult, conversion to a book-style presentation delayed the release of the information and also limited the number of fields that could be included. The PDF version of the Cities Annual Report only provides a small fraction of the data SCO collects from reporting cities.
In September, SCO made the transition to open data by publishing the city, county and pension system data sets on Socrata’s Open Data Portal. SCO’s open data website is called “ByTheNumbers” and can be found at https://bythenumbers.sco.ca.gov/. In early December, SCO added special district data and refreshed the city and county set with fiscal year 2014 statistics. This update is especially important because until fiscal year 2012 (which ended on June 30, 2012), city and county data were not published until at least 14 months after the end of the fiscal year. This delay has now been reduced to five months, most of which is accounted for by the time given local governments to submit their reports to SCO after year end.
Bulk data sets available on the site go back to fiscal year 2003, enabling users to evaluate relatively long time series of data. Roper hopes to extend the data back further, recalling a recent incident in which a researcher requested county data from 1978. The question triggered a mad search at SCO for a dusty old volume that fell apart when it was placed on a flatbed scanner. If all these old books were scanned and digitized, the full time-series would be available to anyone on demand. I suspect that Google could help SCO backfill the data set, but, if not, the optical character recognition and table recognition tools listed at http://pdfliberation.wordpress.com might come in handy.
ByTheNumbers is the latest of three open data web sites SCO has created. The first, http://www.publicpay.ca.gov, provides compensation data for state and local employees. State Controller John Chiang decided to build the site after learning about a scandal in Bell, Calif. The Los Angeles suburb with a population of about 38,000 residents was paying its Chief Administrative Officer $787,637 annually – about 30 times the city’s per capita income. Since the site’s launch, it has had eight million hits. While SCO’s PublicPay site anonymizes employees (showing employer and title, but not employee names), a privately managed web resource, Transparent California, adds names to the salary and benefits data.
Chiang’s second foray into data transparency came after California voters passed Proposition 30 in 2012. This measure imposed a temporary tax increase, with proceeds largely earmarked to restore previous cuts in K-12 education and community college funding. At http://trackprop30.ca.gov, users can see how much Proposition 30 revenue has been allocated to a school district and how the funds are being spent. The site also provides links to community college district audited financial statements; links to K-12 school district audits will be added in the coming months.
In a previous Sunlight post, I explained the value of these audited financial statements and called on the federal government to publish those that it collects. It is nice to see California state government setting an example in this regard. I hope that they will add city, county and special district financial statements to the universe of audits they publish. In the meantime, California readers can find many city and county audited financials on my web site at http://www.publicsectorcredit.org/ca – built, in part, with a Sunlight OpenGov Grant.
As the Controller’s office shifts from John Chiang – who becomes State Treasurer in January – to Betty Yee, SCO staff is reaching out to the public for ideas on how to leverage its open data. Last week, SCO announced a “build-a-thon” – basically a month-long, off-site hackathon – in which participants are being asked to build apps on top of the newly released special district data set. Prospective hackers can find the rules here and the data set here (Excel 2007-13 workbook, 76.2mb).
With the help of Socrata’s Open Data Portal, the state of California has taken an enormous step in the direction of making local government readily available to the general public. I hope SCO continues to build out this capability and that other states follow California’s lead.
We are thrilled to be the recipients, once again, of Cards Against Humanity’s generosity. But beyond feeling grateful for their contribution of $250,000 to generally support Sunlight’s operations (a rare thing in this day of grants that exclusively fund specific projects), we are excited Max Temkin and company used this opportunity to raise awareness of how much money pervades America’s political system.
Beginning on the ninth day of Cards Against Humanity’s “Ten Days of Kwanzaa or Whatever,” people who bought their special set of holiday cards received a long scroll identifying who had contributed to their senators—on double-sided paper! And those clever folks at Cards Against Humanity didn’t want to leave folks out who live in places like D.C.—where residents pay taxes but are not represented in the U.S. Senate—so these recipients received a print-out detailing the campaign contributions given to incoming-Senate Majority Leader Mitch McConnell of Kentucky. We think he’d appreciate the paper disclosure, since he’s never been much of a fan of passing the Senate Campaign Disclosure Parity Act, a bill that would mandate electronic filing of campaign finance reports for timely transparency.
We’ve watched with great delight as CAH’s mystery gift package recipients shared their surprise at receiving an itemization of the campaign contributions their senators received. And if you want to know what motivated CAH’s donation to Sunlight, check out their typically satirical infographic.
Who knew understanding campaign finance could be so much fun? And speaking of fun, we hope you’ll check out the song and video CAH commissioned from the very-talented Jonathan Mann, known as the “Song a Day” guy on YouTube. He positively transformed some of Sunlight’s talking points into a very catchy tune. I bet you’ll find yourself singing along during the rest of the holidays.
Information about money in elections needs to be open and transparent — this is a mantra you’ve heard before. But, here’s something new: in Japan, there may not be enough money in elections to make them competitive.
In Sunday’s elections, the Liberal Democrats (LDP) retained their majority in parliament by a landslide, securing another four years in office for Prime Minister, Shinzo Abe. However, the sweep of the self-proclaimed referendum of the current economic policy, “Abenomics,” doesn’t necessarily indicate a glowing endorsement of either the policy or the leadership. If anything, the victory was borne of a lack of opposition and voter apathy. Not only were parties caught completely off guard by the snap elections, but the rigid election structure in Japan makes it difficult for parties to even compete for seats.
After only two years in office, Shinzo Abe dissolved the Diet, Japan’s lower house of parliament, and announced that elections would be held on December 14. This surprise left opposition parties scrambling and ultimately, the second largest contending party, the Democrats, only fielded 198 candidates compared to the LDP’s 352. Without ample time to prepare, opposition parties were already left at a distinct disadvantage to the leading party — and that’s before any campaigning even began.
The structure of political finance regulations in Japan can also make it difficult for opposition parties to gain a foothold into the political system. As we’ve seen in other countries, major political donors are not likely to fund new parties with little influence on policy. So, these parties often rely on public funds to get started. In some places, like Albania, the state even provides “start-up cash” to new parties. In order to eligible to receive public funding in Japan, parties must already hold seats in parliament and have received at least 2% of the vote in a recent election. The amount of media airspace allotted also decreases when a party puts forth less than 12 candidates. These provisions give little promise to parties working to get off the ground.
It’s clear that these rigid regulations are designed to make campaigns more fair by giving parties an equal opportunity to showcase their platforms — and certainly, there’s merit in that — but it also can reinforce the power of existing positions. In exit interviews, those who actually showed up to the polls on Sunday cited a lack of laudable alternatives as the main reason they cast a vote in favor of the Liberal Democrats. Perhaps if there was an opportunity for new ideas to infiltrate the political system and to use campaign methods fit for 2014, more people would have shown up in the first place.
Photo credit: Architect of the Capitol. Today’s meeting of the Legislative Branch Bulk Data Task Force produced some major news. Specifically, there will be a lot of new bulk legislative data in the new year. The newly renamed Government P… […]
In a move that could make it easier for the public to identify who’s behind TV advertising aimed at influencing elections and legislation, the Federal Communications Commission on Thursday proposed requiring cable and satellite television pro… […]
The Federal Communications Commission has begun asking the public to weigh in on whether radio, cable and satellite TV operators should report political ads the way that broadcast TV already does. But for a clear demonstration of just how bad existing data is, the FCC can look to a U.S. Government Accountability Office report released earlier this year.
The government’s spending watchdog gave up even trying to track radio ad spending by issue, concluding there just wasn’t a credible source covering 2007 through 2012.
The point of the report, which was released in September as “Broadcast Television and Radio: Disclosure Requirements for Broadcasted Content” was relatively obscure: track down advertising aired by TV and radio stations promoting their own interests. Though the stations disclose political ads aired by others, they aren’t required to report ads they air on their own behalf, or to air ads with opposing viewpoints. Many TV stations, however, do disclose their own in-house ad buys–but the GAO was unable to say how many because the records are such a mess, and can be shredded after just two years.
Instead, the GAO ended up paying $30,000 in taxpayer funds to Kantar Media, an ad-tracking company, for records of TV ads on topics investigators thought would be relevant, according to a copy of the contract requested from GAO. Only the TV data was good enough to use–the report gave up on even tracking radio at all “due to limitations in public and private data sources.”
The first obstacle to using the public data on TV advertising was the time the GAO wanted to examine. The report set out to look at ads aired between 2007 and 2012. But broadcast TV files need only be kept for two years. Moreover, an order to collect these files from the biggest stations in the country began only in August of 2012; prior to that these files were only kept on paper at individual station offices.
When GAO researchers looked further into the issue, things looked even worse. The TV files are required for political issue ads “relating to any political matter of national importance, including a national legislative issue of public importance.” But there’s no official guide as to what issues meet this criteria. The GAO report noted “stakeholders we spoke with, including FCC and NAB, told us that broadcasters are responsible for making determinations on a case-by-case basis.”
Even after paying $30,000 for data on ads in four subject areas, GAO still couldn’t produce reliable cost estimates of the ads. That’s in spite of the fact that broadcast TV stations make public the cost of every political ad spot they air–something their lobby, the National Association of Broadcasters, has complained bitterly about.
For radio, even less data is available, and the GAO was unable to even find cost estimates. The best information again came from Kantar Media’s Campaign Media Analysis Group (‘CMAG”)–but it had holes. “We determined that CMAG’s radio data were not comprehensive enough for our purposes because these data cover fewer stations in fewer markets than the television data and do not include cost estimates, among other reasons.”
It was another year full of encouraging news on the open data front in states and municipalities across the country. New open data policies were approved in municipalities of all sizes from coast to coast, existing open data programs matured and sparked new innovations, and there were numerous other open government wins as a result of advocacy efforts. Here’s a look back at the year’s highlights.
A growing number of strong open data policies
2013 was a landmark yearfor the adoption of open data policies, and 2014 built on that momentum. There are now 50 open data policies at the state and municipal level across the country. There were 19 new state and municipal policies adopted this year, and four existing open data policies were updated and improved. All five of the largest cities in the U.S. now have open data policies, and small and mid-sized cities are increasingly joining the movement for open data. Bloomington, Ill., with a population of fewer than 78,000; Jackson Mich., with a population under 34,000; and Williamsville, N.Y., with a population of fewer than 6,000, all adopted open data policies in 2014, showing that smaller communities can and should embrace the benefits open data can bring.
The communities adopting open data are finding ways to build on the success of those who came before them in this space. Boston’s new open data policy, for example, encourages the Chief Information Officer overseeing implementation to work with people across departments and issue additional guidance, building a strong foundation for moving forward with buy-in across government. Pittsburgh’s new open data policy follows more than half of the best practices for open data.
Even governments with open data policies already in place are seizing the opportunity to iterate and make improvements. Washington, D.C., the first local government we identified as having an open data policy, updated its policy this year with an executive directive from the mayor. The city made several important improvements, and though it still has plenty of room for growth, we hope it will continue to iterate its policy to include more best practices.
We also examined limited liability donations and corporate registry access in Texas and Washington, finding that states vary drastically when it comes to how they deal with disclosing information about campaign donations.
Beyond financial data, we explored other specific kinds of open data. We took a look at the progress of open legislative data at all levels of government around the country. Opening up legislative data is helping people access more information about proposed bills and existing law, empowering them to be more informed participants in government processes.
Our thought-leadership in all of these areas and beyond has helped share knowledge, inspire advocates, and empower change at the local level around the country.
Building resources to support the movement
Increased open data adoption and improved open government practices have come about through research, advocacy, and the sharing of best practices. Sunlight has developed and continues expanding on a wide variety of resources to help with these efforts.
Building on the foundation of the Guidelines, the Open Data FAQ page answers specific questions about open data policy drafting and implementation, aiming to help the processes be as informed as possible.
The US City Open Data Census, a collaboration between Sunlight, Open Knowledge, and Code for America, is another resource for staying informed about open data around the country. It tracks the technical openness of datasets in cities where a non-government community member has volunteered to do that assessment.
It’s been a great year for open data and open government across the country, and we believe there is always room for improvement. We built upon the great progress made in 2013 to advance open data in states and municipalities of all sizes, helping them craft and implement policies that fit the local context, and we look forward to continuing and growing that work going ahead.
We’re sure 2015 will bring further advances for open data and broader open government wins. There are sure to be more challenges, too, and we’ll be ready to take those on, working with government staff and officials as well as community advocates and stakeholders to bring greater transparency to local governments.
Despite all of the different promises made by governments to publish public sector information or make government data open by default — whether through the Open Government Partnership, the G8 Open Data Charter, the EU Digital Agenda or country-level commitments — many transparency and open data groups around the world are struggling with governments that are reluctant at best, obstructive at worst.
In many contexts, it seems that governments know more about their citizens than citizens know about their administration. One important step forward in resetting the default to open is to create public, comprehensive listings of all information holdings; however, we have seen very few examples of strong index and audit requirements so far. Recently, the government in the Netherlands has welcomed the advocacy efforts of the Open State Foundation for a government-wide data inventory and has agreed to publish an index in 2015.
The flow of information has changed
If you think that information collected and paid for by the government belongs to us all, think again. Despite the launch of new open data projects, initiatives and portals — and despite various types of government declarations and action plans — the unlocking of public service information is not happening fast enough. One might wonder how many more research reports on the economic value and societal benefits of open data need to be published.
Information still flows from government to citizens, from professors to students and from media to consumers. At the same time, there are generations that do not need to be told what is good for them. They can investigate, publish and organize themselves. The way they interact with authority is completely different from previous generations, but the structures of government have not changed yet.
Information is a right, not a privilege
To reverse this, one needs to understand not only how government works, but also how to influence it. Open data allows people to see how their governments operate; public officials need to stop building applications and start sharing the data they hold. When governments build apps, they continue to dictate what people need to know from their own perspectives. This makes these government bodies not necessarily more transparent or open.
The call for open data is an attempt to confront a closed system from the outside. Open data raises questions about public institutions, how they work, how they deal with information and their public tasks. It raises questions about what data is being collected, used, maintained, managed and released. From outside the government, it is difficult to know what information the government holds. Often, public officials first want to know what you will do with the data and exactly what the economic or societal benefits you will create for them. Some governments have invested in research to help them decide what data to open. Others have organized speed dating sessions between the private sector and some government data holders. However, access to government information is a right, not a privilege for the included few.
Let us know what you have
A year ago, we sent a letter to all government departments in the Netherlands, explaining the need to unlock government information and the benefits of machine-readable, government-wide data. We are often asked which datasets we would like to see opened. So, despite well-publicized benefits, there are two major unknowns: We do not know what data the government holds, and we do not know all possibilities and opportunities that occur when particular datasets are opened.
In order to help prioritize what datasets can or cannot be made public, the first step is to know what type of information the government collects and produces. Therefore, we asked all departments to create an data inventory and publish the results, preferably in a machine readable format. Ronald Plasterk, minister of Interior in the Netherlands, who also serves as the coordinating minister for open data, agreed to have such a government-wide data inventory, sending a recent letter to the Open State Foundation confirming that the Dutch government sees open data is an important policy instrument. The results will be published in the first quarter of 2015. We realize this is only the beginning, but it still represents a major milestone for both our country and the whole open government agenda.