Algorithm detected Ebola outbreak nine days before health authorities using internet posts

There's a great article over at TechRepublic by Lyndsey Gilpin on how the computer algorithm behind HealthMap detected the recent Ebola outbreak nine days before it was identified by health authorities.

In How an algorithm detected the Ebola outbreak a week early, and what it could do next, Gilpin describes how by tracking, collating and corroborating information published in online news sources and social media, the algorithm was able to identify the 'mystery hemorrhagic fever' over a week before official health agencies.

However the significance of the outbreak was not realised by HealthMap founders until after health authorities got involved.

This type of use of internet 'chatter' and algorithms to make sense of the world offers enormous potential for organisations to better identify and understand underlying trends.

For government this means the ability to identify outbreaks of human, animal and crop diseases earlier, detect early indications of potential crises and trends in population views and behaviours.

In all these cases it gives government the opportunity - using only public sources of information - to react sooner and more appropriately, containing problems and getting ahead of issues.

Equally this capability can be used by commercial entities for marketing and product development, by financial organisations for faster and better informed investment decisions and by activists, lobbyists, foreign interests and terrorists to identify weak points for destabilising a nation or gaining advantage.

It remains early days in this area - not as early as when Google first released its flu map for Australia back in 2009 - but early enough that few organisations are, as yet, investing in this area (giving them a huge advantage over rivals).

However with HealthMap's algorithms now successful at screening out over 90% of unrelated information, the value of using this type of approach in policy and service delivery has now reached the point of commercial viability, which should only accelerate investment and research into the area in coming years.

If Australian governments aren't yet mining the public internet for intelligence to help improve decision-making, hopefully it won't be long until they do - at least to contend against others who might use this intelligence for less than positive purposes.

Read full post...

View summary...

Government agencies need to think open first with all content - example of the Clean Energy Regulator

Last week the Clean Energy Regulator released a calendar that illustrates when other government agencies use National Greenhouse and Energy Reporting data.

Called the National Greenhouse and Energy Reporting publication calendar, this is useful data for locating government reports on energy and climate change. It also serves a secondary role in highlighting the importance of the information collected and released by the regulator.

Now a calendar, by its nature, is simply a table of data - so it would make sense to release this calendar as data. Indeed as it is a public document, with no security or private constraints, it is a perfect candidate to be released as open data.

This would allow the calendar to be mashed up with other data on the topic to present, perhaps, a comprehensive calendar of climate change and energy research in Australia.

Indeed it would likely be simpler to release this calendar as data than as a formatted document, which would require additional formatting and conversion steps. This could also meet all government accessibility requirements, as well as making the data easily reusable by others.

So what did the Clean Energy Regulator do?

They released the calendar only as a DOC and a PDF.

*deep sigh*

It's clear to me that there's still a major disconnect in government regarding when and how to release data in an open way.

This is likely an education gap, but also a KPI gap. If public servants were required in their KPIs to ensure that relevant public content they were responsible for was published in an open and machine-readable fashion we might see some change.


Essentially agencies need to embed 'open thinking' at the start of their reporting and research processes, working from the basis that all data that is being released publicly - including content such as calendars, lists, financial accounts and more - should be available in a reusable open format.


In this case I've 'liberated' the data for the Clean Energy Regulator and let data.gov.au know, as I did recently for ACT Crime Statistics data.

In this case I've even improved the data by turning the month field into a working date, fixing the errors (where closing brackets were dropped), separating out web addresses as a new field and separating Department/Agency name from the note that follows it, thereby allowing Department/Agency to be analysed and grouped. (view it at https://docs.google.com/spreadsheets/d/1xlnfd0H9t4O1JzdNqhjfeUCw7Oxk1XN7DYM_WbAHZ-c/edit?usp=sharing)

I've also done some analysis on the number of reports by agency and month (as below).



This is the type of work that individuals like me should not be doing.

It's what agencies and individual public servants need to take responsibility for - particularly when opening up the data is actually simpler than locking it down into a less open form.

The National Greenhouse and Energy Reporting publication calendar is now (unofficially) available as a Google spreadsheet for reuse. The Clean Energy Regulator is welcome to take a copy and use it for their publishing updates.

You'll find it at:
https://docs.google.com/spreadsheets/d/1xlnfd0H9t4O1JzdNqhjfeUCw7Oxk1XN7DYM_WbAHZ-c/edit?usp=sharing

Read full post...

View summary...

My presentation from DrupalGov: How open source is powering government

I've attended DrupalGov today in Canberra. Below is my presentation for people who missed it.

There's also a recording that will become available in due course.

How open source is powering government from Craig Thomler

Read full post...

View summary...

Why should governments continue to control voting systems and processes?

Having centralised systems for voting is the standard approach for countries around the world.

In most places it is simply accepted that the government funds the system for election and referendum voting - funding the polling places, ballot boxes, officials and vote counting systems, whether this be directly or at arms length via a body independent of government, but reliant on government funding.

And let's face it, voting is integral to governance. Voting provides legitimacy to a country's government, providing some form of mandate for a ruling party and ensuring that populations are satisfied with a given set of representatives by giving them a role in choosing them.

Looking at it cynically, having governments control voting could be seen as a conflict of interest - the politicians with an interest in re-election both create the electoral laws and fund the system for casting ballots.

Indeed in some parts of the world systematic electoral fraud is a major concern - the government can influence elections outcomes by changing the legal requirements for voting, adjusting electoral boundaries, place onerous condition on forming or operating new parties or on standing for election, limit electoral donations or advertising by opposition parties, or restricting physical access to ballot boxes.

That's before getting to issues with who votes, how many times and how the votes are counted.

In countries where there's substantial trust in governance and the electoral system these issues are generally small-scale, though ever present as we continue to see with voter identification laws introduced in some US states, major parties voting themselves more electoral funding (as Australia's two major parties tried to do in 2013) and individual examples of bad practice by candidates across all democracies.

In places where democracy is fragile and institutions are weak these issues are magnified, and various systems have been developed to keep elections honest - independent observers are often involved (where allowed) to scrutinise an electoral process; citizens and activist groups have photographed and published issues at ballot boxes online via mobile devices, first in ad hoc ways and then via map-based reporting systems such as Ushahidi; entire websites dedicated to exposing electoral fraud and bad practice have popped up around the world.

These systems have often migrated back to established democracies, for example, the mobile phone tool used to scrutinise the 2007 Kenyan elections was reused in the US Presidential race in 2008, demonstrating that in sustaining freedom to vote, eternal vigilance remains important.

However these are simply systems to scrutinise how governments run elections, rather than independent voting processes. They watch and report what happens in electoral systems, but don't seek to replace these systems directly.

Switzerland is perhaps unique in that it has an entrenched system of direct democracy which allows citizens to overrule parliament through a plebiscite vote - but even then the electoral process is funded and managed by the state.

More recently we've seen pseudo-electoral systems emerge - online petition systems like Change.org, which is having a material impact on government decisions. We've also seen systems that allow citizens to put forward laws to parliaments using banking details to validate individual supports (voters) for a given legislative proposal.

Governments broadly keep these systems at arms length, retaining the discretion to ignore these votes where they choose, for whatever reason they see fit - and fair enough, these systems are often flawed electorally, representing specific groups, can be prone to some level of gaming and don't have the same level of scrutiny as a formal government-run electoral process.

However the technology now exists for this to change - and it already is, beginning in Hong Kong.

In June this year two legislative steps by China were seen in Hong Kong to weaken the 'One country, Two system' approach that the city had been operating under since reunification with China.

As a result academics and citizens of Hong Kong started the ‘Occupy Central with Love and Peace’ campaign, which involved the non-violent occupation of the main business district of the city with the goal of achieving universal suffrage for voting in time for the 2017 election of the next Hong Kong Chief Executive.

Attached to this process was an unofficial city referendum which took place from June 20th – June 29th 2014. The poll asked two simple questions: which proposal for universal suffrage would you like to see implemented in Hong Kong and should the legislative council adopts an universal suffrage system if it does not abide with the international definition?

This was held outside (and without the support) of Hong Kong's government by citizens, involving online, mobile and physical voting at 20 'pop-up' polling booths set up across the city, with all Hong Kong residents aged over 18 eligible to vote.

While there were official efforts to prevent the referendum, including a large scale attack on the referendum website, the confiscation of voting boxes by Chinese officials and censorship of mentions of the referendum online by Chinese authorities, these did not prevent large scale voting by citizens.

At the end of the ten day process, 798,000 residents had voted - over 20 per cent of the eligible population. Most had voted via the mobile apps, with the second most popular way being online.

Despite the turnout, the Hong Kong government took the view that civil referendums had no legal standing under Hong Kong law, and therefore the result could be ignored.

This led to the largest public protest in Hong Kong since 2003, with over 500,000 people taking to the streets on July 1st 2014.

A good article detailing the process in detail is at Free Speech Debate, as Vote for Hong Kong – on the streets and online.

This type of unofficial civil referendum, where citizens get together to develop robust electoral systems and use them to state a view to a government, is possible today in much of the world.

The notion that voting systems are the province of governments, that only a central jurisdiction can manage a fair national electoral approach, simply no longer holds true.

So while citizens may choose to allow governments to manage these systems, it is feasible to outsource them - on a case-by-case or a permanent basis, detaching electoral processes from the individuals and groups seeking power.

In the future we may see more populations hold their own civil referendums on government policy or on who governs them.

While governments might decry these as illegitimate, as they are not covered within the laws that parliaments have created, these civil electoral processes may indeed be more legitimate in the long run - as the voting process and system are not designed or modified at the whim of those who hold power.

Indeed it will be interesting to see how the government of an advanced democracy reacts in the face of a civil referendum. Even if they deny the legitimacy of the process, they may find it hard to ignore the democratic backlash.

Read full post...

View summary...

Who won GovHack's National prizes - analysis

I've done some work analysing who actually won the National awards at GovHack based on the official results at govhack.org/2014-winners/, and this is what I've discovered...

There were 87 National GovHack awards given out to Projects and Teams, with one GovHack subcategory (Best Science Reporting) unawarded. Of these, 66 awards were awarded to Teams for their Projects, and 21 to Teams as team achievement awards that were not for a specific project.

Every GovHack location except for Mount Gambier won at least one national award.

Fifty six different teams won awards (across the entire eighty seven awards), with the leading prize winner being Sarbii from Perth with five awards, followed by Jonathan and Wai, Michael de Hoog and R3K1 on four awards and another six teams winning three awards. Eighty per cent of winning teams won either two or one award.



Forty five projects won an award (across the sixty six prizes for projects). Show the Gap was the top awarded project, with four awards, followed by eight projects with three awards: CancerMash, Data-by-region comparator, Energy Calculator and Comparison tool, Sarbii - Search and Rescue, Stat.Map, The Hack Report, What is Gov (Baby don't hurt me) and When the Heck am I?

Again about eighty per cent of winning projects won one or two awards.



Looking at locations, Canberra was the biggest winner by number of prizes (24), followed by Sydney (17), Adelaide (13) and Perth (11).


By the number of prizes relative to the number of entries, noting that some entries won more than one prize so this overstates the actual share of entries that won prizes, the winning location was Tasmania (89%), followed by Canberra (77%), Sydney (68%) and Perth (61%).

The most prizes were awarded in the Team category (22), followed by Best Social Inclusion (Hack (14), Best Business Hack (13) and Best Digital Humanities Hack (12).



You can see all of these statistics and more, as well as links to all the winners, in my Google spreadsheet: https://docs.google.com/spreadsheets/d/18H4gWSuVLb5Mjt84YdymeXSlSaAxjY78lX5T90i6RmQ/edit?usp=sharing

There were also several prizes given to government agencies which I've not analysed:

• Best Government Participation, won by the South Australian Government, with 2nd place shared by the Federal Department of Communication and The Australian Institute of Health and Welfare; and
• Highest Voted Government Data, won by the National Library for Trove and 2nd place going to the Victorian Building Authority.

I've not analysed local awards, which are visible or linked from the bottom of the GovHack winners page.

Read full post...

View summary...