Is there really an open data El Dorado?

I was reading a tweet yesterday from Australia's CTO, John Sheridan, and it raised an interesting question for me.

#opendata a new goldmine? RT @theeconomist: Making official data public could spur lots of innovation econ.st/111Wy4x #gov2au
— John Sheridan (@sherro58) May 19, 2013

Is government open data really a new goldmine for innovation?

The Economist's article, A new goldmine, makes a strong case for the value of open data through examples such as GPS, the Global Positioning System which is owned by the US government (who owns the satellites), but has been provided free to organisations around the world since 1983.

I've also seen fantastic studies in the UK and Australia talking about the value in releasing public sector information (PSI) as open data, and great steps have been taken in many jurisdictions around the world, from Australia to Uruquay, to open up government silos and let the (anonymised) data flow.

I agree there's fantastic value in open data; for generating better policy deliberations and decisions, for building trust and respect in institutions and even for stimulating innovation that leads to new commercial services and solutions.


However I do not believe in an open data El Dorado - the equivalent of the fabled city of gold - where every new dataset released unveils new nuggets of information and opportunities for innovation.

Indeed I am beginning to be concerned that we may be approaching a Peak of Inflated Expectations (drawing on Gartner's famous Hype cycle chart) for open data, expecting it to deliver far more than it actually will - a silver bullet, if you will, for governments seeking to encourage economic growth, transparency and end world hunger.

Data is a useful tool for understanding the world and ourselves and more data may be more beneficial, however the experience of the internet has been that people struggle when provided with too much data too quickly.

Information overload requires humans to prioritise the information sources they select, potentially reinforcing bias rather than uncovering new approaches. Data can be easily taken out of context, misused, distorted, or used to tell a story exactly the reverse of reality (as anyone closely following the public climate change debate would know).

Why assume that the release of more government data - as the US is doing - will necessarily result in more insights and better decisions, particularly as citizens and organisations come to grips with the new data at their fingertips?

A data flood may result in exactly the reverse, with the sheer volume overwhelming and obscuring the relevant facts, or the tyranny of choice leading to worse or fewer decisions, at least in the short-term.


The analogy of open data as a gold mine may be true in several other respects as well.

The average yield of a gold mine is quite low, with many mines reporting between one and five grams of gold per tonne of extracted material. In fact gold isn't even visible to the naked eye until it reaches 30 grams per tonne.

While several hundred years ago gold was easier to find in high concentrations and therefore easier to extract - leading to many of history's gold rushes - over time people have mined most of the highest gold concentrations.

Extraction has become laborious and costly, averaging US$317 per ounce globally in 2007.

There is definitely gold in open data, value in fresh insights and innovations, opportunities to build trust in institutions and reduce corruption and inefficiency in governance.

However if open data is at all like gold mining, the likelihood is that the earlier explorers will find the highest yields, exploring new datasets to develop insights and innovations.

By the gold mine comparison we are currently in the open data equivalent of the gold rushes, where every individual who can hoist a line of code can dig for riches as a data miner, while data analysis companies sell spades.

Following the analogy, data miners will shift from open data site to open data site, seeking the easy wins and quick insights.

However as the amount of open data grows and most of the easy wins have been found, it will get more expensive to sift increasing amounts of data for fewer insights, requiring greater and greater investments in time and effort to extract the few remaining nuggets of 'gold'.

At that point many government open data sites may become virtual ghost towns, dominated by large organisations with the ability to invest in a lower yield of insights.

Alongside these organisations, only a few tenacious data mining individuals will remain, still sifting the tailings and hoping to find their open data El Dorado.

Read full post...

View summary...

Register for the May Gov 2.0 event in Canberra

I've realised I'd not yet blogged about this month's Gov 2.0 event in Canberra, which is being run by the Department of Finance and combines the Gov 2.0 crowd with the Australian Government's  Cross Agency Social Media Forum.

The event - which is coming up next Thursday, 23 May at DEEWR's auditorium on Marcus Clarke street, features four speakers on social media in the public service:

• Tom Burton from the ACMA to discuss his work and strategy;
• Evan Hill from PM&C to present about the APS Policy Visualisation Network;
• Felicity Lawrence from ACT Government to present about her PhD research project on social media in the public service across Australia; and 
• Pia Waugh from the Department of Finance to briefly present about the APS online engagement courses Finance are running. Please see below for more information.

For more information and to register visit the EventBrite page at: http://casmmay2013.eventbrite.com/

Read full post...

View summary...

How governments in the US and UK are using crowdfunding

Delib UK has taken my thoughts on crowdfunding within government and researched a number of other examples where local governments are using innovative ways to engage citizens in paying for communal facilities.

Worth a read at How councils are crowdfunding community projects.

Read full post...

View summary...

Can an 'open' government site be open if it is poorly designed?

I was chatting with Paul Davis on Twitter recently about the The State Decoded, an open source US-developed platform for accessibly and openly exposing state legislation online (see the platform in use at Marylandcode.org).

He suggested that the tool was effectively a US version of Austlii, which is a repository for Australian federal and state law.

My view was that there were significant differences between the two approaches.

The State Decoded is an open source platform being crowd developed, which anyone can replicate for any jurisdiction. It contains APIs, presents all content as accessible web pages and is nicely designed to be easy for casual users to access.

Austlii, on the other hand, is a closed platform developed by two universities. There's no APIs, much of the content is available only as PDFs and documents, and the design - well, minimalist is possibly the right term, with the site difficult to navigate for all but university professors (who developed it) and lawyers.

When I made this comparison (in brief given Twitter's character limits), Paul said to me...

@craigthomler interesting, so design is a determinant of whether an initiative is considered open government.
— Paul Davis (@davispg) May 11, 2013

This made me think a little - do I consider visual design a criteria for openness in government?

And my answer was:

@davispg IMHO yes. A technically open site unusable for casual users due to inconsistent, inaccessible & generally poor design isn't open.
— CraigThomler (@craigthomler) May 11, 2013

I thought in this post I would expand a little on my view.

For some technically orientated people design can be an afterthought. Their focus is on making a system or machine work as it should, able to take in data and spit out information correctly and quickly.

For these people, design is a 'nice to have' added towards the end of the process, with sites and systems made 'pretty' to appease the communications and marketing people, but is otherwise non-functional.

I've participated in many IT-led 'design' processes, where the focus was on how entities within the system should interact with each other, and the testing focused on 'user-acceptance' - which basically is designed to answer the question 'do the system's features work as intended?'.

In these processes there was little or no consideration regarding the visual appeal of the solution, whether the terminology was understandable to the audience, the search results expected or the navigation logical for non-experts and non-programmers. At best there was some commitment to making the site accessible - however this often meant 'bare bones' lists of text on a white background, rather than using alternative methods  to provide a pleasurable experience for all users.

Of course it is essential that websites and system respond quickly and as intended. However if users don't find them appealing, intelligible or intuitive, they will use them unwillingly, if at all.

I like to compare this to the car market. Originally cars were designed to be functional only - with little in the way of 'frills' to appeal to the public. The hard part was in getting the mechanics to work right and to last and car developers (blacksmiths, bicycle and train makers) weren't concerned about appeal.

Today, however, you'd be hard pressed to find any car maker who doesn't strive for visual perfection as much as for mechanical perfection.

Yes we expect cars to perform flawlessly, but we also expect them to look good. All things being equal (mechanically and safety wise), more attractive cars outsell less attractive cars, people develop more attachment to them, use them more and stick with the brand.

So to with products on supermarket shelves. In many cases people are selecting between products which differ little in their composition (or they don't understand the technical differences), simply choosing on the basis of how the packaging looks and makes them feel. Companies build their brands around their visual and emotional connection with customers, with ingredients a secondary (though still important) consideration.

So it is for software and websites. Well designed software systems and sites attract more use - even where they may be technically inferior (who can tell if a site is a few milliseconds slower than a competitor).

And so it is for open government sites. It is certainly possible to make an open government site with brilliant functionality and the best data - however if it doesn't visually resonate with the audience, if it isn't appealing for them to explore and use, it won't be broadly used.

Governments who seek to be open should recognise that it isn't simply about exposing lots of data, or opening the doors for user participation on a mass scale online. Design must be core to the thinking, how sites are designed, how users interact with the system, the structure of the language and of the navigation.

For openness to succeed in attracting broad interest and active participation from citizens, governments must not only think about what they release, how they release it and how they invite citizens to participate.

They must equally consider the citizen-experience, whether citizens can access information or participate in an intuitive and comfortable way, how citizens feel when using the site - excited, engaged and empowered (for a well-designed site), or frustrated, marginalised and stupid (for a poorly designed site).

Design is important and needs to be involved from the start of the development process. How people should feel when engaging should help drive the features and their operation, rather than trying to 'retroengineer' a clumsy system to meet user needs (a far more expensive and unsatisfying process).

So I stand by my view on open government - a technically open site that is unusable for casual users due to inconsistent, inaccessible & generally poor design isn't open.


Indeed, if a government is only playing lip service to openness (forbid the thought), poor design might be an effective tactic to hide things 'in plain sight', reduce the number of user and 'tick boxes' without revealing anything they are required to publish, but don't want easily found.

So where a government, or agency, releases poorly designed open data or engagement sites (particularly as a second or third version), just as they may release a 'bad news' media release under cover of a major news story, or an old report deep in their site (so they can say it is public even though no-one can find it), citizens really need to consider whether there really is a government commitment, or simply the appearance, of openness and transparency.

Read full post...

View summary...

Clinging to a comfort zone

Everyone has a comfort zone. Whether it is a favourite book/movie/restaurant that you return to again and again, that old pair of shoes you'll never throw out because you've worn them in, or the route you take to work each day, we all like to follow familiar patterns, and avoid potentially unsettling, discomfortable, change.

So to in the work place we like to cling to what is familiar and known to us, the systems and procedures established many years ago and the communication channels we know well. Our comfort zone affect the types of careers and jobs we choose, and the approaches and techniques we use to execute them.

This pattern-based approach is employed for sound biological reasons. The human brain consumes 20-25% of our metabolic energy, enormously out of proportion with its size.

Following a routine requires less active thought and therefore less exertion. Thinking is hard work and, as organisms, the tendency is to minimise thinking in order to conserve energy. That's why the more tired people get, the harder it is for them to think clearly or of new things, and why you can accidentally drive home instead of to a friend's house, following your routine.

In other words, moving outside our comfort zone is hard work. We can no longer rely on the known and familiar, we must develop new strategies, identify new risks, consider new opportunities - deal with change and uncertainty, using more energy and creating stress on our systems.

Coping with change becomes even harder and energy-consuming when it is imposed on us outside our control, when events or other people force us outside our comfort zone against our will.

In many cases people resist the change, because habits and routine are easier. Even when the world has changed many people attempt to cling to the past, denying or shutting out the changes in order to continue to exist in a comfortable (and lower energy expenditure) state.

So what does this have to do with Government 2.0 - well everything really.

Government 2.0 represents a set of changes to how government employees engaged with citizens, and how citizens engage with government.

Over the last sixteen years I have seen all kinds of views and behaviour adopted by otherwise intelligent and good people to preserve their status quo - even in the face of overwhelming and highly public evidence to the contrary that the media and public engagement environment had changed, and they needed to change with it.

From denial ('social media is just a fad'), to dismissal ('social media isn't going away but it is only for young people'), to active opposition ('we can't use social media because of these thirty year old rules') - across government and companies alike.

Unfortunately some of this resistance to reality still exist, not because people are bad people, but because they are clinging to their comfort zones.

People such as community engagement professionals claiming that they would never use online consultation because 'face-to-face is best', even while acknowledging that their public events attract few citizens, most being retired.

People in IT teams who want to do everything in a specific software platform, rather than using user-centric, much better and sometimes thousands of times cheaper cloud-based solutions, because they are familiar with the software and prefer costing the organisation time and money to investing their own energy in thinking about new solutions.

People in Communications and Marketing teams who still raise reasons as to why they could never use online  channels to engage citizens and customers, 'we don't know if our audience is online', 'we don't know which tools to use because they keep changing', 'we don't understand the risks' and 'we don't understand the technology'. Isn't it their job to learn what communications options available to their organisation so they can pick the most appropriate for their goals?

Ultimately, however, these individuals will be swept aside as the world keeps changing and the nature of work changes.

Today we see well-developed social media teams in organisations that didn't have a social media channel five years ago. We see agencies reshaping their processes and services to suit online channels, the Victorian government gradually adopting a 'mobile-first' strategy, the UK government a 'digital first' approach.

In the US the President has just issued an executive order requiring all agencies to make all data open and machine-readable by default, while appropriately protecting privacy and confidentially. The order also requires all agencies to publish a list of all the data they could make open but that they, as yet, haven't - an 'open first' strategy for data (watch video below featuring the US Government's CTO and CIO.



The mandates from governments in the UK and US will force more agency staff from their comfort zones. The change programs they employ will help individuals make the changes with minimal energy expended on thinking (most has been done for them).

In Australia we're a little further behind, largely grappling with guidance and policies rather than instructions and mandates. However it is my view that this will change, that governments in Australia will soon follow overseas leads to mandate openness for agencies, not just recommend it.

Is your agencypreparing for this change? Designing and placing the systems, support and training in place in your agency to facilitate it?

Or is your agency clinging to its comfort zone, with senior management secure in the knowledge that such a change could never happen, or if it happened, your agency could ride the storm with minimal impact, or even oppose it because your data is too sensitive/commercial/private/valuable/worthless for it to be mandated for release?

Read full post...

View summary...