Tuesday, January 10, 2012

Why aren't Aussies using open government data to create value?

This post was inspired by a comment by John Sheridan on Twitter,
craigthomler what I'd like for  ? egs of use of  for service delivery innovation, value creation etc, not just curiosity
It's a good New Years wish and highlights two questions that I have been pondering for a long time.

1. Why aren't people making more use of publicly release government data?
2. Does making government data publicly available have any value if people aren't using it to value add?

Let's take them in order...

1. Why aren't people making more use of publicly release government data?
In Australia the data.gov.au catalogue contains 844 datasets (and growing). NSW (data.nsw.gov.au) and Victoria's (data.vic.gov.au) catalogues are also quite large. 

By comparison, the US data.gov catalogue contains over 390,000 datasets, Canada's data.gc.ca over 265,000, the UK's data.gov.uk around 7,700,  Singapore's data.gov.sg about 5,000 datasets and New Zealand's data.govt.nz over 1,600 datasets.

Across these six countries (I am excluding the two states), that is in excess of 670,000 datasets released publicly. However if you search around there's not that many apps listed using the data. The US site lists around 1,150 and Australia's site lists 16 - however that's not many compared to the number of datasets.

As Victoria's data blog asks, what has happened to all the apps produced in government-sponsored competitions? Are they actually worth holding?

OK, let's work through a few possibilities. 

Firstly it could be that these datasets are being widely used, but people simply aren't telling the catalogues. Data may be embedded in websites and apps without any communication back to the central catalogue, or it may be downloaded and used in internal spreadsheets and intranets. In this case there's no actual issue, just a perceived one due to lack of evidence.

Secondly, to face facts, the majority of people probably are still not aware of these data catalogues - they haven't really been widely promoted and aren't of much interest to the popular media. Therefore there may be hundreds of thousands of people wishing to access certain government information but unaware that it is readily available.

Thirdly, those people aware of these datasets may be daunted by the number released, unable to find the data they specifically want to use or simply aren't interested.

Finally, perhaps simply releasing a dataset isn't enough. Few people are data experts or know what to do with a list of values. Could it be that we need simple and free analysis tools as well as raw data?

There's steps governments can take to address all of these possibilities.

If people aren't telling the government about their apps, why not establish light 'registration' processes to use them which capture information on why they are being used? Or if this is too invasive, offer people appropriate incentives to tell the central catalogue about their uses of the data.

Secondly, there may be a need to promote these data catalogues more actively - to build awareness via appropriate promotion.

Thirdly, perhaps we need to do more user-testing of our data catalogues to better understand if they meet the audience's needs. Combined with excellent mechanisms for suggesting and rating datasets, this could greatly inform the future development and success of these catalogues.

And finally, governments need to consider the next step. Provide the raw data, but also provide sites and tools that can analyse them. Sure governments are hoping that the public will create these, and maybe they will, however that doesn't mean that agencies can't do so as well. There's also pre-existing tools, such as Yahoo Pipes, IBM's Manyeyes and analytics tools from Google which could be pre-populated with the government datasets, ready for users to play with.

Alongside all these specific solutions, maybe governments need to start using some of the tools at their disposal to ask why people aren't using their data. Is it the wrong data? Presented in the wrong way? Too hard to use? Market research might help answer these questions.

2. Does making government data publicly available have any value if people aren't using it to value add?

Now to take the second question - does it really matter whether people are using open government data anyway?

Are there other goals that releasing data addresses, such as transparency and accountability, intra-government sharing and culture change?

If the mandate to release data leads to government culture change it may drive other benefits - improved collaboration, improved policy outcomes, cost-savings through code and knowledge sharing.

Of course it is harder to provide a direct quantitative link between releasing data, changing culture and the benefits above. However maybe this is what we need to learn how to measure, rather than simply the direct correlation between 844 datasets released, 16 apps created.


  1. I think one of the key things data users want is consistency. It was a key theme at both Canberra Barcamp and Govcamps over the last 12 or so months. Currently their is limited standards in how data is delivered onto data.gov.au making it a much more difficult task to build cosnistent interpretation tools to handle different data with different subject matter. People dont want to have to write code simply because similar data comes from different sources. Maybe this has changed in the last six months?

  2. So - a lot of the time, I don't have the capacity to hack on the data :(

    I do find really neat datasets occasionally. Usually, I'll push them into freebase - ie: list of business locations of government agencies.

  3. Why aren't Aussies using more government data sets? Heaps of reasons but the main themes for me would be:

    1. Low semantic maturity of Australian government data sets. How much of the data is machine readable? How much is in meaningful formats and non-propiretry? How about being able to download the data in excel and a shape file? Or maybe even using kml?

    2. Finding data. Data is often poorly tagged and difficult to find. Heck, half the states don't even have a www.data.mystate.gov.au web site but instead rely on AGIMO to prop them up.

    3. Data Integrity. Much of the data is out of date or is split into several data sets instead of one single source of truth.

    4. Data availability: When governments release data then take it offline after a competition has run its course is it any wonder that developers become gun shy of exploiting released data?

    5. Data volume. 800 data sets of which several are actually the same? Not good enough. If I want to carry out an analysis on road accidents and the locations of radars over time then I should be able to - but that data does not exist online (or if it does then I haven't found it) in Australia. by having a small amount of data then the value of Open Data for researchers will remain very low.

    6. Lack of munincipal data: Its all well and good to have data from states and the commonwealth, but I expect many would like to know about their local community. Its appear to me that some of the most successful open data programs have been run by a city - not by a State, let alone a federal - jurisdiction

    I've seen some data sets from the ACT for example currently use Stromlo projection for geolocations. Wouldn't lat/long make more sense?

    I agree with @grmsn that consistency is important as well and that standards need to be followed and at this stage the governments of Australia appear happier to be seen to do something in the Open Data arena rather than to actually do something. Releasing datasets in an ad-hoc manner which don't follow standards is an indicator of this IMHO.

  4. I agree with many of the points raised here. Indeed there is an overwhelming number of reasons why app developers haven't been able to JFDI.

    I would add to the "Data availability" point by saying "Don't count your gov2 apps before your open culture has hatched.". I have scoured the .gov.au domain many times to find exactly the data I want available for all the internet to see, only to have absolutely nothing happen when I write to the department (and sadly AGIMO/data.gov.au) to get permission to use it in a serious application. If you try to release the app anyway, suddenly a lone public servant will magically appear to get it taken down by threatening Google/Apple/your ISP with copyright infringement and cyber-crimes (been there, done that).

  5. I'd be very interested to hear the details on "Anonymous' " concerns as detailed in the previous comment. I haven't seen any evidence of the sort to which Anonymous refers. If he/she is willing to email me (john.sheridan at finance.gov.au), I'd be happy to review it.


    John Sheridan
    Agency Services Division, AGIMO