Make government data freely available

An interesting article was released in the Yale Journal of Law & Technology earlier this year discussing a view that government should focus on providing usable data online rather than full-blown websites.

Titled Government Data and the Invisible Hand , the premise was quite simply explained in the abstract:

Rather than struggling, as it currently does, to design [web ]sites that meet each end-user need, we argue that the executive branch should focus on creating a simple, reliable and publicly accessible infrastructure that exposes the underlying data. Private actors, either nonprofit or commercial, are better suited to deliver government information to citizens and can constantly create and reshape the tools individuals use to find and leverage public data.

This approach is very much at odds with the current approach both in the US and Australia, where in most cases the respective governments provide both the data and all the interpretation designed to meet the needs of specific audiences.

Via the current approach, data can becomes difficult to extract, or is presented in a way that is not useful. On that basis these websites are difficult to use. They are also expensive to develop and maintain and difficult to keep current.

The approach in practice

I've encountered both approaches in Australian government websites.

In a past role, managing the website for a private sector water and energy utility, one of the consistently most trafficked areas of our website was local weather. This section had only a peripheral involvement with the main focus of the site, however the level of usage made it important to retain.

We did not run this weather service ourselves. Instead we used a raw data feed provided by the Bureau of Meteorology (BOM) for free. The data was simply customised and represented in an attractive way in our website.

Ours was not the only organisation using this data - a number of other organisations had built businesses though providing weather information - sometimes combined with video, maps, commentary or other feeds. These sites collectively attracted more traffic than the BOM itself.

To my recollection, provided this data was not packaged and directly resold commercially, the BOM had a policy of giving away the data freely.

This approach helped ensure that the public were able to access accurate information, to the public good. It is important to note that BOM data was collected and processed by people and equipment already paid for out of the public purse.


On the other hand, the Australian Bureau of Statistics (ABS) provided a great deal of the data I used in my day-to-day role.

This data was preprocessed by the ABS into tables or excel documents. These were often chunks of information that were not much use to my audience.

My team spent many hours manually deconstruct and reconstruct the ABS data into different forms to make it useful for our corporate needs.

The ABS did not provide data as a raw feed. While the ABS did gave away its data for free online - and this was fantastically useful - the overhead that went into its website inevitably made it less timely, therefore reducing its value in a commercial sense.

So, in comparison:

BOM

• Gave its data away for free online (public access to public data)
• No data analysis required (lower cost to the agency, faster to market)
• Referrals from everyone reusing the data (reach)
• Enormous innovation in how the data was 'mashed' with other sources / analysed and presented (lower cost to agency, transference of risk of misinterpretation to private sector)
  

ABS

• Gave its reports away for free online - but not the raw data (public value but less timely)
  
• Provided intensive data processing (quality assurance but higher cost to the agency, slower to market)
• Limited online reuse, therefore fewer referrals (lower reach)
• No innovation in data analysis and presentation (higher analysis/presentation cost to agency, any risk of misinterpretation stays with the agency)

In my view BOM's approach seems to be both lower costs and risks for the agency and delivers greater public benefits, greater data use, innovation and agency reach (referrals).

Bt the way, it's worth pointing out that BOM is the most trafficked government website in Australia. ABS, despite a wider range of statistics, is much further down the list.


Can the data approach be used across other agencies?

I believe it can. Even in my agency we release numbers and resources which could be indexed and provided in a raw data form for reuse.

We also have a website estimator for calculation purposes. There are around a dozen similar estimators that do a similar job - several providing virtually the same result as the official estimator. However those 'fan' estimators cost nothing to the public purse to create or maintain.

So if members of the public are prepared to create these tools, why should the agency?

Granted this last example is a little more tricky than that - the estimation process is time-consuming and maximising accuracy is a key goal.

However there are other government websites and tools which could and would be delivered by private organisations and individuals, if only the government allowed access to the data stream.


Level the playing field

Note that the article does not suggest that government should stop analysing data and presenting this analysis in websites.

What it suggests is to provide the raw data on a level playing field, thereby allowing private and public organisations the same capacity to use it.

The best way to ensure that the government allows private parties to compete on equal terms in the provision of government data is to require that federal websites themselves use the same open systems for accessing the underlying data as they make available to the public at large.

This means that government agencies such as the ABS can continue to provide reports for people who choose not to do their own analysis.

However it opens the field to innovation and the use of various data sources to make connections that government, in a siloed form, is not as able to do.

This levelling is critical - if the government wants to see innovation it should not hold back the 'secret sauce'. The data needs to be available in a way that allows private and other public enterprises to use it in an equal way.

Open systems are available today via standards such as XML and RSS - look at how Google syndicates maps and ads or how Facebook allows the creation and dissemination of applications.

In conclusion

Government has a crucial role to play in the collection of data across the country. This is a task well suited to the public sector as it is in the public interest that this be available.

However government doesn't have the systems or culture to be best suited to interpret and combine this data or make it useful for individuals and organisations.

Government should provide interpretations - however it should not hold an artificial monopoly over this.

By allow other organisations to access the raw information innovation in its presentation can occur more rapidly, providing deeper insights for the public good.

Make government data freely available.


Does anyone have other examples of where government collected data has become freely available? I'd love to blog about the successes.

Read full post...

View summary...

The online impact of the US election - implications for Australian government

I've previously blogged about the impact of the online channel on Barack Obama's campaign and how it contributed significantly to his nomination as the Democrat presidential candidate.

There have also been broader implications for the US government scene, as captured in a Pew report released last week.

Pew has been one of my favourite commentators over the last six years due to the down-to-earth nature of their reports on online usage. I take their analysis as a prediction of where Australia will be in the next two to four years.

Their latest report, The Internet and the 2008 Election, surveyed normal Americans on their engagement with the 2008 US presidential election via the online channel.

What it found was that 46% of Americans have used the internet, email or SMS to get political news and share their thoughts about the campaign and 23% say they receive emails urging them to support a candidate or discuss the campaign once a week or more.

Now those might not sound like high percentages, but there are a couple of things to keep in mind.

Three elections ago (in 1996 - when Bill Clinton became US President), those figures would have been virtually zero. That's the speed at which the landscape is changing.

Secondly, keep in mind that only 60.7% of Americans eligible to vote actually did vote in the 2004 presidential election - and this was the highest percentage since 1968.

While given that Americans who do not vote may still follow the elections and receive emails about it, the following figures do take on much greater significance in light of the number of 'active voters' in the US.

Pew found that 19% of Americans go online once a week or more to do something related to the campaign (one-third of 'active voters'), and 6% go online to engage politically on a daily basis (ten percent of 'active voters').

It also found that 10% of Americans (fifteen percent of 'active voters') use email to contribute to the political debate and that 1 in 10 of those using SMS (4% of the adult population) are sending or receiving text messages about the campaign or other political issues on a regular basis.

Obama supporters were much more likely to use online communications. They outweighed Clinton supporters by 74% vs 57% and McCain supporters by 65% vs 56% (comparing online supporters that have gotten political news and information online).

Statistics aside, what does this mean for government?

The first implication is that online is now an important channel for electioneering. The last big shift in media use by politicians was more than forty years ago when Kennedy trumped Nixon in the first television debate - signalling a shift in politics from voice to image.

The online shift means that different values become important. Image will remain important in politics (in today's consumer-driven society how could it not), but consistency, substance and depth are also becoming critical.

When citizens can read or listen to a candidate's speech and instantly check their voting record and comments over the past ten years (such as via OpenAustralia) it becomes very clear to the voters when politicians are modifying their positions and dishonesty becomes a critical issue.

This isn't really new - one of the first well known instances was in 1998 when the Lewinsky scandal regarding President Bill Clinton was broken by the Drudge Report.


A second maor implication is on public sector management and governance. It's not only the histories and stories about politicians that become available online, it's also the performance records of government agencies and top public servants.

Where there are close ties between politicians and ministerial offices, politicians can be judged by how their departments engage and conduct themselves online. Individual senior public servants may also find their public comments or lack thereof also coming under intense scrutiny in a political sense.


Does this mean that government agencies and public servants need to hide under their shells and say as little as possible online?

I don't think so.

Firstly, this approach would not work. Citizens are very capable of creating their own websites, transcribing or recording comments and press releases and republishing or referring to them to demonstrate real or contrived inconsistencies in positions or behaviours.

Secondly it's not a bad thing to be held for public scrutiny. Without this scrutiny there's no point to having a democratic process. Politicians and public servants should be held accountable for their views and positions and, to some extent, their past choices.

Finally, online discussion is a benefit to getting accurate information into the public eye.
Once a conversation begins it becomes possible to contribute to it, clarify the issues and ensure that an accurate view is visible. This does require substantial agility - online conversations occur in real time and government doesn't have the time to consider, reflect and rework a statement over weeks before making comments. There is the need for rapid responsiveness, often within hours rather than days.

Read full post...

View summary...

Follow-up on real-time government - video stream available online

A quick addition to my post on real-time government, here's more background on how online technologies are being used to enable real-time government.

Watch Congressman's Culberson video stream for an explanation of an approach he's using to reach his constituents as a 'real-time representative'.

Read full post...

View summary...

Why should government care - how many people use Firefox/Safari/Opera anyway?

I've noticed a tendency in Australia for government agencies to focus on having their websites work perfectly in Microsoft Internet Explorer, but not always quite so well in Firefox, Safari, Opera or other web browsers.

This isn't limited to the public sector, private sector organisations face the same issue of cross-browser compatibility.

On one hand there is the Australian Disability Discrimination Act 1992 and a suite of other standards which relate to web accessibility, as well as state accessibility laws, that organisations - particularly government departments - are required to comply with.

On the other hand there are resource and testing limitations which constraint what organisations can do.

It is clearly important for organisations to support as many web browsers as possible, as the fewer the browsers supported, the longer the tail of people who will not get a satisfactory experience on a website.

There's also the possibility of legal risk. There's already been a high profile court case in Australia on accessibility, regarding the Sydney Olympics (for a great analysis of it by Joe Clark see Reader’s guide to Sydney Olympics accessibility complaint)

Applying web standards is te obvious approach, but not always the simple solution. The standards are quite complex and open to interpretation. Even when your web professionals believe you've met the W3C standards there can be variations in how your site will display in different 'standards-based' browsers.

Specifying which browsers you support is another approach. Simply choose the most used browsers and support those, with custom style sheets to address any page rendering differences. This will catch a good 95% of the market in the top four or so web browsers, but leave a tail of users with older web browsers or less well known products who may not receive the same experience.

So what is the best solution to ensure your organisation meets accessibility standards, delivers the intended experience and doesn't bankrupt itself in the process?

Unfortunately I don't have the knife to cut through for this Gordian knot, every organisation needs to weigh the considerations and decide its own best path.

I can provide a few further references to feed this decision.

Links to various accessibility legislation and guidelines are listed above. Most states in Australia also have government web standards they apply which can provide some guidance on the topic.

As for browser market share, below is a chart detailing the latest share figures from a major statistics collector.

Alongside outright browser shares, it is critical to consider web browser versions as well. While Internet Explorer has a 74% share (down from over 95% before Firefox was introduced), of this roughly 47% use IE7 and 27% use IE6 or earlier. Similar splits also occur for Firefox and other browsers.

Another good source for browser usage is your own web logs, which can provide a more audience specific view of who accesses your website. My agency uses Webtrends to analyse this data, but the majority of web log analysis tools will provide similar information.

Web browser shares - Q2 2008
Source: Wikipedia - Usage share of web browsers

Read full post...

View summary...

Real-time government - not as far away as you think

I was able to take part in an inpromptu live online video and text discussion this morning between Congressman John Culberson of Texas and initially with the Managing Editor of the Wall Street Journal, and then with Grover Norquist, the President of Americans against Tax Reform (pictured below) at a fund raising event in the US.

While I'm not their constituency (although I am a Director of an oil company with production interests in Petrolia), there were a number of others watching who were, and it was a fascinating look at the democratic process in evolution.

The online discussion was an opportunity for Congressman Culberson to directly reach and represent his constituency, asking and answering questions put to him.

The entire discussion was managed through the use of free online social media technologies, Twitter was used to inform the Congressman's constituents that the discussion was taking place and then Quix was used to capture the video on a mobile phone and stream it live to the internet , presenting it on a page where people could text chat directly with the participants, receiving verbal replies.

As one of the first in I was privileged to be mentioned by name - making it the first time I think I've been mentioned by an elected US Congressman.

That type of closeness - between individuals and their elected officials - is now becoming available, and we'll see how soon these technologies are in use in Australia's community Cabinet meetings and similar events.

Below is a screenshot I took of the event (see me at right logged in as CraigThomler).

The Congressman will also be conducting several further interviews through the evening using the same tools and the permanent record will be up at Quix.

Read full post...

View summary...