Craig Thomler's professional blog - AI and digital government thoughts and speculations from an Australian perspective
Craig Thomler
I've worked in the online sector since 1995 in roles including founder, publisher, journalist, webmaster, marketer, channel manager, CIO, COO and visionary. I left the public sector in early 2012 to lead Delib Australia as Managing Director Australia and New Zealand. More...
In conjunction with the Data.gov launch, Sunlight Labs in partnership with Google has launched the Apps for America 2 competition to find the best mash ups using US Federal government data.
Over US$20,000 is being given away for open source online applications using the Federal datasets at Data.gov based on the following criteria:
Transparency: Does the app help citizens see things they couldn't see before the app existed?
Permanence: Will the app be usable over a long period of time? Does the idea have survivability?
Design & Visualization: Does the app look great? Does the app visualize data in a new and interesting way?
I wonder what innovation the Australian government could unlock if it made public data available in machine-readable formats, then held its own competition.
The US government has taken the first step towards meeting President Obama's pledge to make US taxpayer funded information freely available online in reusable formats late last week with the launch of the Data.gov website.
At launch the site featured dozens of datasets from around 20 Federal agencies ready to be used by the public, commercial and NGO sectors in mash-up applications and services.
The public is also able to suggest additional information to be made available through the site.
I am an extremely big fan of making public sector data available online (where there are no security issues), particularly when the data is readily available for online reuse through APIs, XML, RSS, KML/KMZ and similar machine-readable formats.
A speech by the US Federal CIO, Vivek Kundra launched the site and made the purpose extremely clear in the video below. Kundra expects more than 240,000 datasets to eventually be available online, per the post at Governing People by George Fahey, Data.gov opens.
It takes a long hard look at the opportunities for the public sector in using online media to engage the public, arguing that,
In a world where sharing information is so easy, public agencies should aim to improve citizens’ ability to access information, provide feedback, and view the feedback of other citizens. Ultimately, we should move toward a world where the background information, the thinking, and the options for every public sector decision and action are easily available to citizens.
There are numerous case studies of activities being taken by governments around the world - even one from Australia.
It's a good read for senior executives in the public service who are looking to understand how the world is changing - and the opportunities and risks for their departments for embracing the changes versus hanging back.
Information held by the Government is a national resource and should be managed in the public interest. Access to government information increases public participation, and leads to increased scrutiny, discussion, comment and review of government activity.
It would be great to hear an Australian government make a statement like the above, acknowledging that much of the information collected by the government should be readily accessible.
The next step is to make information available in an appropriate format for easy reuse online, allowing it to recombined and used in innovative ways that add value and lead to new insights.
In the US President Obama's newly appointed (and first) Federal Government CIO Vivek Kundra has committed to finding new ways to make government data open and accessible.
He also said he hopes to use emerging technologies like cloud computing to cut the need for expensive contractors who often end up "on the payroll indefinitely."
These are not idle words from a political appointee - Kundra, who I have mentioned previously, is well-known amongst egovernment practitioners around the world for his innovative work in pushing the boundaries of egovernment as the District of Columbia's CTO.
Politicians often have reservations about releasing raw data, despite being collected using public funds, due to perceived concerns that the data might be used to politically damage their reputations.
Similarly government departments often restrict the release of raw data due to concerns over how it may be reused or misused.
In Australia we even go to the extent of copyrighting government data. In the US most data, publications and other tools created by their Federal government are copyright free.
However with the US's moves the debate will soon shift to the disadvantages of not allowing free access to most raw government data.
As history has recorded, countries that remove barriers to the free flow of ideas and information develop faster, are economically more successful and their people enjoy higher standards of living.
Fostering innovation directly leads to national success.
So in a world where some countries make data freely available, how do other nations continue to compete?
To draw an analogy from the publishing world, Wikipedia disrupted the business model for Encyclopedia Britannica. By providing free 'crowd-sourced' information of greater depth and about the same accuracy as a highly expensive product, Britannica has been struggling to survive for years.
After trialing a number of different protective business models to sustain its existence, but protect its data, Encyclopedia Britannica has finally adopted one that might work - it has opened its articles up to 'crowd-sourcing', accepting suggestions which are then reviewed and acted on by its professional editors - a step towards openness. Visit the Britannica blog to learn how to suggest changes to the encyclopedia.
In other words, you cannot beat openness with secrecy - the only way to remain successful is to step towards openness yourself.
This really isn't news. Many have talked about the need for greater openness of government data before. I've even mentioned it myself once or twice.
To finish, I thought I'd flag this recent talk given by Tim Berners-Lee (the father of the world wide web) at TED on the need for open data. It has some points worth reflecting on.
The US government has taken the first holistic step towards making federal data available online in bulk.
This would allow the public to reuse the data, mash it up with other sources and repackage it online in innovative ways - potentially uncovering relationships and new uses that were never conceived of by the government.
The step is being enshrined in the new Omnibus Appropriations Bill, which is already in the process of being passed.
In the words of the sponsor for the feasibility study,
“In our web 2.0 world, we can empower the public by providing them with raw data that they can remix and reuse in new and innovative ways," says Honda, who is vice chairman of the Appropriations Subcommittee on the Legislative Branch. "With these tools, the public can collaborate on projects that can help legislators to create better policies to address the pressing challenges facing our nation.”
The Australian government has a great deal of data which could similarly be freed up to work for the common good of the nation, but access to it varies by department and there are no common standards for how it should be made public.
Perhaps we also need a centralised legislative approach to move towards greater and more consistent access.
NSW has been the scene of an interesting series of events over the last week regarding the right of iPhone developers to republish NSW Rail timetables in their applications.
Why copyright material that is made available online anyway? if the aim is to simply prevent out-of-context use or commercial reselling, there are options like Creative Commons available.
Indeed the ABS has taken steps in this direction, beginning to publish most website content under a Creative Commons license.
I envisage timetables as an appropriate type of information to be offered as a web service or RSS feed. This would allow NSW Rail full control over the accuracy of the information while allowing other websites and mobile applications to integrate it into their offerings. The public can then vote with their feet as to which version of the information they prefer, and how much they are willing to pay for it.
Much the same type of service is already offered by the Bureau of Meteorology - and the success of OzWeather on the iPhone is a testament to the successful integration of information possible when there's a web-savvy government agency and a developer who finds a way to add value to the raw data.
Dave Fletcher has published a post on his Government and Technology Weblog about how the community is making use of Utah government traffic data to create innovative tools to help drivers.
With traffic being a key concern for residents of most big cities, and given the success of the Australian Bureau of Meteorology in allowing mash-ups of weather (another big citizen concern), I wonder when we'll see state governments in Australia allowing this form of use of their traffic data.
As Dave concludes, due to allowing community mashups of government data,
The end result is that Utah government data actually has much broader disemmination and utility than just what is presented on the state's website. With hundreds of information systems, there are many more examples where this kind of sharing could be valuable, but it doesn't happen. It's a big opportunity.
Located at www.diabetesmap.com.au), the service is a wonderful example of how online map systems (in this case Microsoft's free Virtual Earth tool) can be 'mashed-up' with data to provide new ways of visualising and understanding data.
Governments collect and distribute a massive amount of public data each year. It is a continual challenge to make this data accessible and usable for citizens, commercial organisations, researchers, scientists and policy makers.
This challenge isn't limited to a few dedicated statistical organisations, such as the ABS. Many other government departments collect, collate and publish extensive public data about their customers, about the market and about their operations.
Putting on my previous private sector hat, public data can be difficult to locate, download and use in a meaningful way to add value to an organisation. I have struggled at times to discover all of the data I needed and combine the different datasets (from different public providers) with internal data in ways that add value to my employers.
Described as a "convenient way to share, access, and use public data", the system is designed to provide "a centralized repository of public data sets that can be seamlessly integrated into AWS cloud-based applications."
Why is this significant? The approach makes it much faster and easier for organisations to locate, download, customise and analyse large public data sets - such as census, scientific or industry data.
Using Amazon's system,
Now, anyone can access these data sets from their Amazon Elastic Compute Cloud (Amazon EC2) instances and start computing on the data within minutes. Users can also leverage the entire AWS ecosystem and easily collaborate with other AWS users. For example, users can produce or use prebuilt server images with tools and applications to analyze the data sets. By hosting this important and useful data with cost-efficient services such as Amazon EC2, AWS hopes to provide researchers across a variety of disciplines and industries with tools to enable more innovation, more quickly.
Amazon has already exposed data sets such as the US Census and various labour statistics. Shortly it will also provide transport databases and economic databases.
All of these are public data sets being provided by US government bureaus.
Also available are scientific information such as Human Genome data, a collection of all publicly available DNA sequences and chemical structures.
Amazon is also working to provide further public domain or non-proprietary data sets and invites organisations to submit applications for data to be included.
Given that this data capacity sits alongside Amazon's cloud computing service, providing an expandable virtual computing environment, it becomes possible for a range of organisations, researchers and individuals to access and make more effective use of large sets of public data, supporting innovation and democratising the marketplace.
It also allows for the creation of data mash-ups, combining data across different agencies with other data sources, maps, graphics, charts and analysis tools to generate new ways of experiencing data and new insights.
I don't expect Amazon to be the only provider of this type of capacity, Google is very committed to cloud computing and organising the world's data. Microsoft and IBM are also moving rapidly into these spaces.
In the long run I see this type of platform as a very valuable distribution tool for governments seeking to make their public data accessible and usable by the broadest possible group of citizens and organisations.
In turn this will broaden and deepen innovation and permit new realisations based on cross-referencing data from different providers - becoming a competitive advantage for countries savvy enough to make their public data more accessible.
What would it take for Australia to make its public data available via this type of channel? A phone call or email to Amazon and some work in structuring our datasets.
That's a low entry cost compared to the challenge of building a replica system.
I've been reading an article in the New York Times regarding the public competition Netflix has been holding.
The competition, named the Netflix Prize, has a prize of US$1 million for the individual or group who can improve their movie/TV recommendations engine by 10%.
The article, If you liked this, sure to like that, discusses how Netflix's programmers had gone as far as they could with their available resources and skills, so the company decided to make a large slice of their information available publicly (anonymised to protect privacy) and see where others could take it.
There are now over 33,000 teams around the world competing to come up with insights and algorithms to improve Netflix's recommendations, with a public leaderboard tracking the top forty (the best is currently at 9.44%) and a forum where the teams collaborate on improving results, sharing tips and code.
I can't help but think about this in the context of government. Every agency struggles to provide the best possible outcomes and customer service with the resources they are given. However few departments or agencies look outside for help - even to other government bodies.
I'm sure there are many complex problems in government that could be looked at in a similar context to the issue Netflix is facing - ranging from simple IT programming issues, to customer service maximisation (such as the most effective placement of face-to-face locations to cover audience needs) and those huge thorny issues, such as devising fair policies or reforming tax regimes.
I wonder if government would be more effective if it allowed talented people to devise potential solutions (for kudos or prize money), which could then be tested, reviewed and the best solutions potentially adopted.
This isn't just a pipe dream. The UK government is running a competition at the moment, asking the public to come up with innovative ways to use government data to add value. The US and Japanese Patent Boards are piloting having the public examine patents and provide views before they are granted and New Zealand had the public write the Police Wiki Act 2007 (on how the police are to act towards the public).
I cannot think of any Australian examples - if anyone know of some let me know.
Clearly there's all kinds of guidelines and governance required for Australian governments to feel 'safe' in inviting outsiders to assist us in improving governance in Australia - but what do we really have to lose?
While many governments around the world pursue the 'one portal' approach, a few commentators are arguing for a different type of model - many correct doors rather than one big door.
This means reaching out to embed government content in the websites citizens choose to visit rather than simply attempting to encourage all web users to go to a single central portal for all government-related content. This approach is described well in the Read Write Web article, E-Government Meets Web 2.0: Goodbye Portals, Hello Web Services, which states, in reference to the online channel that,
Gartner's conclusion is that governments "should make sure that their information, services and applications are accessible through a variety of different channels, some of which are not controlled or directly owned by government."
This is similar to how government agencies already distribute physical publications beyond their own shopfronts - into libraries, doctors' surgeries, lawyers' offices and into the shopfronts of other government agencies. It also reflects how government has a presence at various community and commercial shows, festivals and other events.
In both these cases government reaches out into other organisations' venues in order to better reach citizens in the places they frequent.
I'm a proponent of this 'any door' approach being extended online. As an egovernment practitioner I do not necessarily care how people get to the information and services my agency provides online, provided that they get to them.
This means I am a supporter of central portals as an avenue outside my agency's own website to reach our customers. It also means I am a fan of greater cross-agency collaboration on information provision, where agencies with similar audiences work together to provide government information to citizens.
Most importantly it means I am a supporter of rss, mash-ups and embedded web services - any technology that allows my agency to reach beyond the confines of its own website to reach our customers in whatever websites they choose to visit - commercial, public or citizen-run.
For example (hypothetically), if my agency produced a video relevant to the customers of any organisation involved in the family law system, it would be worth our while to look at how we could reach beyond our own website traffic to the traffic of other involved websites.
Using Youtube, we could generate a video that can be embedded into any site across the family law system, thereby potentially magnifying the reach of its content.
Assuming that my agency has 10 percent of the traffic to the family law system, this could, with the agreement and support of other organisations, result in up to a 10x boost in traffic to the video - all targeted at the appropriate audience.
If we also had the video included in the australia.gov.au portal this would lift usage further, but in a less targeted way, as the portal does not specifically target the same audience as we are attempting to reach.
The approach in this scenario applies for any type of government information distribution online. It also means that government needs to think more about how it provides information online, and how easy and attractive it is for other organisations to embed the information, not just link to it.
Long-serving public servants are familiar with the challenge of balancing customer, agency and political wishes, with a clear understanding that the role of the public service is to implement the policies of the government of the day and not to be customer advocates or lobbyists.
In the commercial sector similar challenges are often faced between customer, management/board and shareholder interests, however often the choice of master to serve is less clear. Without customers a business fails, thereby failing to meet the goals of management or shareholders.
This influence doesn't exist in most of the public sector - citizens are not able to pick and choose their service provider, the government makes that decision for them, providing many services in a monopoly environment.
This monopolistic model has worked well for government over the last century, ensuring that the bulk of citizens have a consistent experience - whether by phone, print or face-to-face.
However the effectiveness of the monopolistic model doesn't fully hold in the online world. Suddenly governments are not the only organisations with universal reach, and suddenly citizens can create services that fill gaps left by government. This has given rise to a multitude of sites where, for little or no cost, citizens are providing better and more cost-effective services than can the slow moving and cumbersome wheels of government.
These citizen-provided services are also totally citizen facing, without any need to answer to political masters, making them often better attuned to community needs.
How should government address the challenge of these 'competitors', by crushing them out of existence (an easy task for legislators)? by ignoring them (as often seems to currently be the case)? or by embracing, supporting and encouraging them?
Personally I feel governments should embrace and support these 'competitors', helping them access government data in order to improve their offerings and aiding them in reaching broader audiences - even at the expense of the government's own sites.
Of course, this willingness to be transparent and collaborative doesn't occur over night - as discussed government agencies are not acclimatised to competition, and the skills of most agencies do not reflect the skills useful in a competitive environment.
However I hope that this changes over time and government begins to support and foster these competitors, learning from them how to better meet the needs of customers.
Here's a video about some of the work of these citizens in filling service gaps that government had not yet either seen a need for, or been funded to address.
In August I analysed traffic to our agency's website in July 2008 using Hitwise's data measurement service, comparing our share of web traffic against the total to Federal government websites, other government websites and the top websites visited by Australians.
The results provided me with a view of how important government websites are in peoples' online lives - not very. Less than 2.5% of website visits were to government sites.
It also helped me form some ideas as to how Australian government departments can make their online channels more effective means of engaging citizens.
Reviewing Hitwise's reports from July 2008, tracking around 2.95 million Australians' visits to over 647,000 websites (using ISP logs), the total government sector (6,634 sites) accounted for only 2.26 percent of all tracked website visits by Australians.
Of these,
Federal government's 2,094 sites accounted for 57.61% of all Government traffic and 1.3% of visits to all tracked websites,
State government's 2,183 sites accounted for 30.82% of all Government traffic and 0.70% of visits to all websites,
Local government's 1,596 sites accounted for 6.51% of all Government traffic and 0.15% of visits to all tracked websites,
The other 761 sites (often foreign government agencies) accounted for 11.4% of all Government traffic.
In comparison, Google.com.au and Google.com together accounted for 9.64% of Australian visits to all tracked websites, four times as much as the total government sector (Google.com.au, the number one site visited by Australians, accounted for 7.85% and Google.com for 1.79% of visits).
Facebook, the fourth most visited website, received 2.36% of total tracked visits - slightly more than the entire Australian government.
MySpace, the seventh ranked site, received 1.78% of total visits - almost 50% more than Federal government sites.
Only one government website regularly reaches Hitwise's Top Twenty list of Australian sites, the Bureau of Meteorology (coming in at 16th position with 0.51% of traffic in September 2008). In fact, this site alone accounts for almost a quarter of the visits to the total government sector.
To put these figures into perspective, I roughly estimated from my Agency's actual web traffic that each Australian web user in July 2008 made 270 visits to Hitwise tracked websites (note that at an average visit duration of 10 minutes, this is significantly less that the figure reported by Netratings in March 2008 (PDF) - of 13.7 hours/week online).
Of these estimated 270 visits,
26.6 visits were to Google,
6.3 visits were to Facebook
4.8 visits were to Myspace
3.5 visits were to Federal government sites,
2.4 visits were to State government sites,
0.4 visits were to Local government sites.
Even if you discount my estimate and take another measure of the average number of website visits per Australian each month, the proportion based on Hitwise's tracked websites remains the same.
What does this mean for government? Even a few visits per month by Australia's estimated 11 million plus regular internet users users adds up to a significant online audience for government in Australia.
However my conclusion is that Australian government departments should not rely on reaching our citizen audiences simply via our official websites.
We need to reach out and engage our customers via the websites they choose to use.
These non-Government websites account for over 97.5% of regular internet usage by Australian (per Hitwise's July 2008 figures).
If Australian government wants to effectively communicate with citizens online, our departments need to invest in understanding where our audiences spend their time, reaching beyond our official sites to engage them in the online communities they choose to frequent.
How do we engage citizens on their own turf? There are many different ways that private organisations reach out to user communities, and government can learn and use many of these approaches, such as
using search advertising to promote Australian government services prominently across top search engines and community sites,
providing web feeds (RSS, Atomic, etc) that other sites can merge into their own information channels,
providing data feeds that can be mashed up into widely used sites and new functions (as the Bureau of Meteorology does so well),
creating, and promoting, useful portlets and widgets on popular platforms (Google Gears, Facebook, Blogger, Ning) that can be added to individual and group social sites,
white-labelling services that can be embedded in other sites (Slideshare, Youtube),
reaching out and participating sensitively in forums, blogs and wikis relevant to our communities,
engaging online advocates and supporting them as we do media representatives (as US consumer goods firms engage 'Mummy Bloggers' and US political parties engage political bloggers),
creating and supporting themed community spaces for citizens (as Huggies has done in Australia for mothers).
Note Hitwise checked the numbers drawn from their web reporting service (thank you Alex and Rebecca). The idea for this post, the conclusions drawn and any calculation errors are mine alone.
Just as organisations are beginning to listen to the conversations about them on forums, in blogs and in online newspaper comments, Twitter has become a powerful and important mechanism for tracking US public opinion during the US Presidential election.
In a custom-built application, Twitter has used its search and trending tools to build an updating commentary on the election, visible at http://election.twitter.com.
Thinking back to the days of tickertape news releases and stock updates, this is a high-tech equivalent reflecting the views of US Twitter users.
With the integration of a mention weighting system and positive/negative indications,it would be possible for any politician or organisation to get a 24/7 view of their public sentiment.
Any sudden changes in the normal flow of comments could then be mined to detect and pre-emptively prepare for issues before they reach the broader media.
For the sceptics, who do not see Twitter as a valid channel for government communication with the public or media, here's a list of US government 'A-list' Twitterers, including the White House, Senator Obama and a selection of State Governors and large US agencies.
And for those who like poetry and the big picture, 3D Twittervision provides an interesting global insight - particularly during major crises.
One of the primary changes resulting from the growth of the internet has been to place professional media tools and distribution capabilities in the hands of individuals.
Any individual with access to a computer and internet connection can create and distribute prose, poetry, commentary, software or services to millions at little cost. Add a microphone and they can conduct talkback or share original music and ass a low-cost digital camera with video capacity and they can also share photos and video.
As detailed in Paul Budde's article, The rebirth of the innovative individual, this is a return to the individual creativity stifled by 'big media' through the 20th century.
The private sector in Australia has already begun effectively tapping into this media change - but how about the public sector? Car, computer and shampoo companies have supported customer-created television commercials. Prominent political bloggers have been invited to political rallies and 'mummy bloggers' courted by large consumer goods companies. Online musicians have hit the top of the charts and been 'discovered' and online programmers have contributed solutions to major corporate software solutions.
We've even seen other media, newspapers, radio and television channels add the capability for individuals to break stories and provide video and audio coverage used around the world.
On the public front, in other jurisdictions we've seen some government efforts to tap into this space, from the New Zealand police wiki Act (an Act of Parliament written by the public via a wiki) to the UK cash prizes for mashups (where the government is rewarding the best applications built using government information).
In Australia I have not seen any developments on this scale.
Certainly we have a few mechanisms to listen to citizens via different online consultation channels, but listening isn't the same as collaborating.
I am not aware of any major initiatives in Australia where government is saying to the public,
We don't know how to achieve the best outcomes in this area, so if anyone has a good idea put it forward and we'll both reward and use the best of them?
This can be challenging step for any organisation used to be the source of answers, rather than the facilitator of solutions.
However, as the private sector is discovering, the new approach delivers excellent outcomes.
I'm hopeful that within a few years we will also see Australian governments using collaborative approaches to write legislation, generate program ideas, produce creative and develop (online) applications and systems.
Over in California a controversy over the level of public access to public information flared up where the The Bee newspaper in Sacremento published a searchable online database of public sector officials and their salaries.
The newspaper simply pulled publicly available information together into a single source - no information was assumed or obtained illegally.
Over time we're also likely to see more Australian public information also being matched in this way - the tools to do so are readily available today.
For instance, it would require minimal effort for a media outlet or individual to mash together and republish information from GOLD (Government On-Line Directory) with APS salary ranges drawn from agency careers pages - providing a fairly accurate picture of the salaries of senior public sector officials in Australia.
Pulling together names of public servants listed in Hansard reports, media releases, websites and from published event attendence lists or meeting minutes, could also be combined with White Pages details, salary figures and office locations to identify and locate many public workers. Add a Google search and you can discover sporting affiliations, past roles, comments made online and further information about individuals - particularly those with distinctive or unique names (such as myself).
Each of the pieces of information by itself doesn't breach privacy - so can matching them together create a breach?
If so, how do you prevent information matching - shut down the internet and close the public libraries?
...everything we know about records management is wrong. Sure, that's hyperbole, but Barton [Founder of Glassdoor, now publishing salary information on the web] isn't exaggerating when he claims, "People's appetite for this information ... is effectively infinite." Once again, the Internet will show us what happens when public records are actually public.
A-Space, an online collaborative space for US intelligence operatives, is planned for launch this month, giving all 16 US intelligence agencies a streamlined and effective tool for sharing information and collaborating - activities that have been criticised as previously lacking across US intelligence initiatives. As reported in FCW.com, in the article, A-Space set to launch this month, after logging in,
analysts will have access to shared and personal workspaces, wikis, blogs, widgets, RSS feeds and other tools. To log in, analysts will need to prove their identity using public key infrastructure, and their agencies must list them in the governmentwide intelligence analyst directory.
Like many social-networking sites, each analyst will create an online personal profile, and colleagues can see what others are working on and the A-Space workspaces that they are using. In addition, much like Facebook, users can also post notes on one another’s profiles
The A-Space social network will include a search tool and data sets from six agencies at launch, with more to be progressively added.
Since the release of the report, The power of information (which I'm planning to blog about next week), the UK government has taken enormous steps towards using the internet in more empowering and collaborative ways to improve service offerings, communication and transparency.
Now the UK government has released a host of new government data, previously unavailable, for use in the development of mashups by individuals, companies and government agencies.
This is a virtually (excuse the pun) unheard of step for any government - to release data in a way that makes it easy for anyone to combine it, analyse it, represent it in combination with other data and form conclusions.
Taking the concept even further, the UK government is stimulating innovation by running a competition for the best mash-ups created using the data, with 20,000 pounds in the prize pool.
Posts
