Australian open data goes CKAN - but remains fragmented by jurisdiction

In the run-up to GovHack this weekend, Australian governments have been hard at work preparing new or updated open data sites, many using CKAN as their base technology.

This includes the new Australian Government beta data.gov.au site, the new South Australian government's data.sa.gov.au and the NSW government's new data.nsw.gov.au/data/.

We also have Queensland's data.qld.gov.au, Victoria's data.vic.gov.au and the ACT's data.act.gov.au (which uses Socrata) at state level.

Tasmania, Western Australia and the Northern Territory have yet to launch open data sites, although I know at least two of these jurisdictions are considering how they begin.

It's fantastic to see Australian jurisdictions opening up their data, as well as the increasing mandate at political levels, highlighted by Queensland's appointment last year of an Assistant Minister for eGovernment, and the recent Australian Government announcement that they were joining the international Open Government Partnership (OGP).

However, let's hold the horses here a moment.

Why are Australian jurisdictions each launching their own open data site when we could have a single whole-of-government site - which would provide easy access to any data, from any Australian jurisdiction, side-by-side?

Why not set common naming conventions for datasets across jurisdictions, common data formats for its release and common field (and field names) in each common dataset?

Unfortunately this fragmented approach is still the reality for Australia - even when there's a commitment to the same direction or approach, various governments prefer to 'go their own way' rather than work together to save costs and improve efficiencies.

Each jurisdiction sees itself as a 'special case' and there's limited capability across governments to coordinate a single solution that wouldn't get mired in politics - at both political and bureaucratic levels.

So while Australia is stepping forward with open data, it's also revealing the issues in our system of government. Cost-efficiency is often trumped by political advantage or concerns that certain jurisdictions are not carrying their 'fair share' of the load.

Given it is unlikely that jurisdictions will solve these issues quickly - what I would really like to see out of GovHack this year is a project that aggregates all of Australia's open data into a single repository, matches it and presents it in a common data format.

This would allow people seeking common datasets from different jurisdictions to find them all in one place and use them easily due to a consistent format, rather than having to go to six or more sites to find the data and having to convert it for use.

Read full post...

View summary...

Australian academia beginning to learn to crawl in 2.0 social channels

I've long lamented the speed at which academia was embracing the internet, social channels and 2.0 approaches - with limited courses available on modern online techniques for under and post graduates, old fashioned-approaches to research and publication.

There's been hints of brilliance overseas - with US universities placing courses online and UK universities embracing social in a major way - however Australia has largely remained a backwater for higher education in a 2.0 world, with individual exceptions at specific universities, such as Dr Axel Bruns and Julie Posetti.

To demonstrate some of the impact of this Australian academic drought, a few months ago I was approached by a European professor about identifying an Australian academic working in the Gov 2.0 field to write a chapter in an upcoming book on Government 2.0. 

This professor, who I had previously worked with on a major report on global Gov 2.0 for the European Parliament (unfortunately not publicly available), had failed to identify anyone in Australia working in the Gov 2.0 space through her academic channels.

I made initial enquiries through a number of my Gov 2.0 contacts in government, as well as to a range of academics and universities, however was unsuccessful at finding anyone through their systems. In the end I was very lucky to encounter an academic in South Australia with relevant expertise at an event I was speaking at in Adelaide. This academic is now working on the book project and I'm very interested in how it turns out.

We have seen some recent stirring towards greater acknowledgement of 2.0 approaches in the recent ARC (Australian Research Council) moves towards open access publishing of public-funded research, however this is still a very small step.

We have also seen some good debates on the role of the public in science, and some pilots such as the Peer-to-Patent, which strike at the commercial end of the spectrum, and the Atlas of Living Australia, which involves citizens in mapping Australia's biodiversity.

We're also now seeing some steps to move beyond the traditional peer review process to consider new ways of measuring the reach and impact of academic research, with the 'altmetric' movement gaining steam.

What are altmetrics? I admit I hadn't heard about them until recently and when I first encountered the turn found the name little more than marketing buzz. 

Essentially the term describes the use of online social metrics to assist in measuring academic success - mentions on Facebook and Twitter, the level of reuse of raw research datasets via APIs, 'semantic publication' of specific passages and references to academic articles in blogs and forums, and more.

The term altmetrics was developed by the founders of one of the first companies that is spruiking altmetrics solutions to academics, and the biggest supporters of the term are other companies seeking to profit from the same rush to web statistics. Therefore I am still inclined to regard the term itself as marketing buzz for the types of social metrics commercial and public sector organisations have been using for years (see the chart below on the growth of use of the term in Google searches).

However it does signify an important and major change in how academic research is measured and valued.

If academics begin measuring their success in how well discussed and commented on their work is in the public sphere, they will likewise begin talking more about their research publicly in order to grow their buzz and their recognised academic prowess.

This will encourage academics to get out from their lecture theatres into the community, become more proficient at communicating their thoughts and work to a broader layman audience and making research more accessible, interesting and influential in public debates and policy work.

I also hope more publicly available research will also lead to more people interested in pursuing these careers, greater commercialisation of research work, improved scrutiny of findings and better social outcomes.

However I hope that at some point academics will realise that 'altmetrics' are simply no more than metrics - ones that are already becoming business-as-usual in commercial and public sector spheres - and focus more on involving people in and sharing their research than on the marketing buzz.

For more information on altmetrics, see:

• A manifesto on altmetrics: http://altmetrics.org/manifesto/
  Also see the noteworthy tools - some have value for open policy work as well.
• A post from a company founded to develop and sell altmetrics tools to tertiary institutions, with good links to information about what altmetrics are: http://blog.plumanalytics.com/post/49874093012/plum-analytics-and-altmetrics-in-the-news

Read full post...

View summary...

Where are all the public sector web analysts?

It's not unusual for agencies to spend millions of dollars on a new program supporting some group in society and hundreds of thousands of dollars on consultants and research to ensure that the new program is designed to be successful, to track its progress and audit it over time.

It's not unusual for agencies to spend tens of thousands of dollars on an online presence to educate people (such as on a new program) - a website that may provide information, eligibility and application processes as well as a couple of social media channels for engagement.

However it is rare for an agency to spend even thousands of dollars on social media monitoring and website analytics to determine whether the website and online presence for a new program is being effective, where improvements could be made, to track its progress and audit it over time.

I've worked in several government agencies which had some kind of research unit, who spent their time analysing customer and program information to provide insights that help improve policies and service delivery. Unfortunately, even in the five and a half years I spent in the public sector, I saw these units cut in responsibilities, reduced in size, even turned into contractor units simply managing external research consultants.

These units were still new to online when I entered the public service, unsure of how to analyse it or how to weigh the insights they might receive. However when I left, although online had been recognised as an important channel, the capability of research units to integrate it into other analysis had been sadly diminished due to budget cuts.

This trend towards outsourcing or simply disregarding data analysis, at a time when society has more data at its fingertips than ever before, is worrying in government. What trends are going unnoticed? What decisions are being made without consideration for the facts?

However I have a special concern around how government agencies regard web and social media analysis, which in my view are increasingly useful sources of near real-time intelligence and longer-term trend data about how people think and behave.

As I've never been able to afford to have a web analytics expert in one of my teams in government, I've spent a great deal of my own time diving into website and social media stats to make sense of why people visited specific government sites, what they were looking for and where they went when they didn't find what they needed.

I've also used third party tools - from Hitwise to Google Trends - to help identify the information and services people needed from government and to help present this information to agency subject matter experts and content owners to help inform their decisions on what information to provide.

I know that some agencies have begun using social media analysis tools to track what people are saying online about their organisation and programs, often to intervene with facts or customer service where relevant, and this is good and important use of online analytics.

I'm even aware of web and social media analytics being provided back to policy areas to help debunk beliefs, much as I used to give different program areas snapshots of their web analytics to help them understand how effective their content was with the audiences they targeted (when I had time).

With the rise in interest in open data, I guess what I'd like to see in government agencies is more awareness of how useful their own web analytics can be to help them to cost-efficiently understand and meet citizen needs. I would also like to see more commitment of resources to online analytics and analysts within agencies to help their subject matter experts to keep improving how they communicate their program, policy or topic to layman citizens.

It may also be a good time to look into the intersection of open data and online analytics - open analytics perhaps?

I would love to see agencies publishing their web traffic and social media analytics periodically, or  live, allowing government websites to be held accountable in a similar manner to how data on crime statistics helps keep police accountable.

Maybe certain web statistics could even be published as open data feeds, so others might mash-up the traffic across agencies and build a full picture of what the public is seeking from government and where they go to get it. This could even allow a senior Minister, Premier or Prime Minister to have full visibility on the web traffic to an entire state or nation - something that would take months to provide today.

This last suggestion may even overcome the issue agencies have in affording, or for that matter finding, good web analytics people.  Instead external developers could be encouraged to uncover the best and worst government sites based on the data and provide a view of what people really want from government in practice, from how they engage with government online.

Read full post...

View summary...

Australia is joining the Open Government Partnership

As reported in Peter Timmins' Open and Shut blog this morning, the Attorney-General, Mark Dreyfus, has announced that Australia will be joining the Open Government Partnership (OGP) - the leading global organisation for states working towards more open government.

Australia is the 59th country to join the OGP, following Ireland (who sent a letter of intent last week) and excluding Russia, who has just withdrawn their application.

It makes us the fifth country in the Asia-Pacific region, behind South Korea, the Phillipines, Indonesia and Mongolia.

This is a timely (if not overdue) commitment by Australia, particularly considering how well-developed our government transparency regime is already. We're ranked 4th in the country list of the Open Knowledge Foundation's Open Data Census (see below).

Now the questions we'll see answered to over the next few years are: how will OGP membership influence the level of attention and mandate for open government in Australia; how will it affect how Australian Government agencies operate, and; will New Zealand follow suit?

Read full post...

View summary...

Is there really an open data El Dorado?

I was reading a tweet yesterday from Australia's CTO, John Sheridan, and it raised an interesting question for me.

#opendata a new goldmine? RT @theeconomist: Making official data public could spur lots of innovation econ.st/111Wy4x #gov2au
— John Sheridan (@sherro58) May 19, 2013

Is government open data really a new goldmine for innovation?

The Economist's article, A new goldmine, makes a strong case for the value of open data through examples such as GPS, the Global Positioning System which is owned by the US government (who owns the satellites), but has been provided free to organisations around the world since 1983.

I've also seen fantastic studies in the UK and Australia talking about the value in releasing public sector information (PSI) as open data, and great steps have been taken in many jurisdictions around the world, from Australia to Uruquay, to open up government silos and let the (anonymised) data flow.

I agree there's fantastic value in open data; for generating better policy deliberations and decisions, for building trust and respect in institutions and even for stimulating innovation that leads to new commercial services and solutions.


However I do not believe in an open data El Dorado - the equivalent of the fabled city of gold - where every new dataset released unveils new nuggets of information and opportunities for innovation.

Indeed I am beginning to be concerned that we may be approaching a Peak of Inflated Expectations (drawing on Gartner's famous Hype cycle chart) for open data, expecting it to deliver far more than it actually will - a silver bullet, if you will, for governments seeking to encourage economic growth, transparency and end world hunger.

Data is a useful tool for understanding the world and ourselves and more data may be more beneficial, however the experience of the internet has been that people struggle when provided with too much data too quickly.

Information overload requires humans to prioritise the information sources they select, potentially reinforcing bias rather than uncovering new approaches. Data can be easily taken out of context, misused, distorted, or used to tell a story exactly the reverse of reality (as anyone closely following the public climate change debate would know).

Why assume that the release of more government data - as the US is doing - will necessarily result in more insights and better decisions, particularly as citizens and organisations come to grips with the new data at their fingertips?

A data flood may result in exactly the reverse, with the sheer volume overwhelming and obscuring the relevant facts, or the tyranny of choice leading to worse or fewer decisions, at least in the short-term.


The analogy of open data as a gold mine may be true in several other respects as well.

The average yield of a gold mine is quite low, with many mines reporting between one and five grams of gold per tonne of extracted material. In fact gold isn't even visible to the naked eye until it reaches 30 grams per tonne.

While several hundred years ago gold was easier to find in high concentrations and therefore easier to extract - leading to many of history's gold rushes - over time people have mined most of the highest gold concentrations.

Extraction has become laborious and costly, averaging US$317 per ounce globally in 2007.

There is definitely gold in open data, value in fresh insights and innovations, opportunities to build trust in institutions and reduce corruption and inefficiency in governance.

However if open data is at all like gold mining, the likelihood is that the earlier explorers will find the highest yields, exploring new datasets to develop insights and innovations.

By the gold mine comparison we are currently in the open data equivalent of the gold rushes, where every individual who can hoist a line of code can dig for riches as a data miner, while data analysis companies sell spades.

Following the analogy, data miners will shift from open data site to open data site, seeking the easy wins and quick insights.

However as the amount of open data grows and most of the easy wins have been found, it will get more expensive to sift increasing amounts of data for fewer insights, requiring greater and greater investments in time and effort to extract the few remaining nuggets of 'gold'.

At that point many government open data sites may become virtual ghost towns, dominated by large organisations with the ability to invest in a lower yield of insights.

Alongside these organisations, only a few tenacious data mining individuals will remain, still sifting the tailings and hoping to find their open data El Dorado.

Read full post...

View summary...